Workspace/Mini projects
Loading progress
All mini projects
MODULE 29 · 5 HOUR BUILD

Serving capacity notebook without a GPU

Produce a reproducible capacity report that combines architecture-level cache math, page reservations, and scenario comparisons, then defines a future measured load test.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Run the seed and verify the byte calculations for the invented architecture.
  2. Add validated configuration input for model weights, cache dimensions, context limits, and headroom.
  3. Compare at least six prompt/output/concurrency scenarios with per-request page rounding.
  4. Add an arrival-trace simulation with queueing, cancellation, and a documented fairness policy.
  5. Write a hardware validation protocol that records quality, cold/warm cache behavior, accepted throughput, latency, failures, and peak memory.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • All reports label arithmetic estimates and simulation results clearly.
  • Every memory field states bytes or a named binary/decimal unit.
  • At least three hand-computed scenarios match the program.
  • A long-output scenario cannot be admitted using prompt-only memory.
  • The report includes a case where free aggregate tokens are insufficient after per-request page rounding.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
import json

GIB = 1024 ** 3
ARCHITECTURE = {"layers": 32, "kv_heads": 8, "head_dim": 128, "value_bytes": 2}
SCENARIOS = [(512, 512), (2048, 2048), (4096, 4096), (17, 0)]

def bytes_per_token(config):
    return (2 * config["layers"] * config["kv_heads"]
            * config["head_dim"] * config["value_bytes"])

def scenario(prompt, output, cache_budget, token_bytes, page=16):
    logical = prompt + output
    reserved = ((logical + page - 1) // page) * page
    per_request = reserved * token_bytes
    return {"prompt_tokens": prompt, "max_new_tokens": output,
            "logical_tokens": logical, "reserved_tokens": reserved,
            "wasted_token_slots": reserved - logical,
            "request_cache_bytes": per_request,
            "estimated_concurrency": cache_budget // per_request}

def build_report():
    device, weights, runtime, headroom = 16 * GIB, 4 * GIB, 2 * GIB, 2 * GIB
    budget = device - weights - runtime - headroom
    token_bytes = bytes_per_token(ARCHITECTURE)
    return {"scope": "invented architecture; arithmetic estimates only",
            "architecture": ARCHITECTURE, "cache_budget_bytes": budget,
            "bytes_per_cached_token": token_bytes,
            "scenarios": [scenario(p, o, budget, token_bytes)
                          for p, o in SCENARIOS],
            "excluded": ["runtime variance", "prefix sharing", "latency", "quality"]}

if __name__ == "__main__":
    report = build_report()
    assert report["bytes_per_cached_token"] == 131072
    assert report["scenarios"][1]["estimated_concurrency"] == 16
    assert report["scenarios"][3]["wasted_token_slots"] == 15
    print(json.dumps(report, indent=2, sort_keys=True))

Push it further

Run the proposed load test on an authorized real serving environment, then explain discrepancies between estimated and observed memory and latency without rewriting the original assumptions.