Build a memory and latency model before buying capacity

Autoregressive serving usually has a prefill phase that processes the prompt and a decode phase that generates subsequent tokens. Time to first token includes queueing, preprocessing, prompt computation, and the initial generation work. Inter-token latency measures the spacing between generated tokens. Total request latency also depends on output length, serialization, network delivery, and any agent tools called between generations. A single tokens-per-second number cannot describe all of these experiences.

Start a performance record with workload shape: prompt lengths, output lengths, concurrency, arrival pattern, model revision, precision, and hardware. A workload with short prompts and long outputs stresses a different part of the system from document analysis with very long prompts. Agent loops add another dimension because one user task can generate several requests. Measure task completion as well as individual model calls when sizing an agent service.

Weight payload is parameter count multiplied by storage bytes per parameter. An invented seven-billion-parameter dense model at two bytes per parameter has a 14-billion-byte payload, about 13.04 GiB. Decimal GB uses powers of 1,000; binary GiB uses powers of 1,024. Mixing these units can turn an apparently safe deployment into an out-of-memory failure. Quantization scales, unquantized layers, runtime buffers, graphs, and allocator overhead add to the payload.

Unlike a training budget, ordinary inference does not need optimizer state or retained backward activations. It still needs intermediate activations and often substantial KV-cache storage. Reserve headroom for realistic concurrent requests and transient peaks. A model successfully loading on a device proves only that its initial allocation fits; it does not show that a production request with the promised context and generation limits can complete.

For a simplified full-attention decoder, cache bytes are 2 × layers × KV heads × head dimension × bytes per stored value × cached tokens across active sequences. The factor two accounts for keys and values. Use KV heads, not automatically the number of query heads: grouped-query attention can share keys and values across multiple query heads. Sliding windows, hybrid layers, compression, and implementation-specific allocation can change this simple model.

Consider 32 layers, eight KV heads, head dimension 128, and two-byte cached values. One cached token consumes 131,072 bytes, or 128 KiB. A 4,096-token sequence therefore needs 512 MiB. Sixteen such sequences need 8 GiB for this cache alone. The token total includes retained prompt tokens plus generated tokens. Prefix sharing can reduce duplicate allocations, but capacity guarantees should not assume a cache hit that the workload may fail to deliver.

A simple bandwidth bound divides bytes that must be read by sustainable memory bandwidth. If an invented decoding step reads 14 GB of weights and memory sustains 700 GB per second, the weight-read lower bound is 0.02 seconds, or 20 milliseconds. This is an idealized bound, not measured token latency. Attention-cache reads, arithmetic, transfers, kernel launches, and imperfect utilization can add time; batching may share weight reads across several active sequences.

A compute bound instead divides required operations by sustainable compute throughput. Compare these estimates to reason about whether a configuration is more constrained by moving values or multiplying them. Peak vendor specifications rarely equal achieved performance for a particular kernel and shape. Small batches may underutilize arithmetic resources, while large batches can encounter cache capacity or latency limits. The model should guide which measurement to collect next rather than substitute for it.

Suppose a hypothetical device offers 16 GiB, with 4 GiB for weights, 2 GiB for runtime allocations, and 2 GiB reserved for uncertainty. The remaining 8 GiB supports at most sixteen of the 512 MiB cache allocations above under this simplified accounting. If each request may grow beyond 4,096 tokens, reserve its promised maximum or implement a documented preemption policy. Admitting only from current prompt length risks failure midway through generation.

Validate the estimate with measured peak memory under representative lengths and concurrent load. Record rejected and queued requests, since a service can preserve latency by silently refusing most traffic. Capacity planning should state the acceptable quality, latency, throughput, and failure rate together. If the hardware changes, rerun the workload rather than carrying over a tokens-per-second claim based only on nominal memory size or accelerator model family.

All architecture and capacity values are invented for arithmetic practice. The estimate excludes paging waste and architecture-specific cache behavior. Change token_count to include prompt plus maximum generated tokens, then observe how available concurrency changes.