Workspace/Mini projects
Loading progress
All mini projects
MODULE 23 · 6 HOUR BUILD

A routing replay and economics report

Build an offline replay tool comparing routing policies on identical tasks. Produce a report of task success, spending, escalation, and latency distribution with every synthetic assumption visible.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Run the synthetic seed and explain why each threshold routes each case.
  2. Define an input schema for task IDs, route outcomes, usage, and latency observations.
  3. Add exact cost accounting with a versioned rate table and explicit missing-data behavior.
  4. Compare direct and cascading routes using the same held-out tasks and a fixed verifier.
  5. Add per-task paired differences, segment summaries, and request-level p95.
  6. Write a recommendation bounded by the observed dataset and document quality or latency constraints that disqualify a cheaper policy.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • Every policy is evaluated on identical task IDs.
  • Total cost reconciles to unique call records without cached-token double counting.
  • All reported percentiles state estimator, sample size, and timeout treatment.
  • Synthetic results are clearly labeled and no real provider prices are invented.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
from math import ceil
from statistics import mean

CASES = [
    {"id": "a", "difficulty": 0.1, "small_ok": True, "large_ok": True},
    {"id": "b", "difficulty": 0.4, "small_ok": True, "large_ok": True},
    {"id": "c", "difficulty": 0.7, "small_ok": False, "large_ok": True},
    {"id": "d", "difficulty": 0.9, "small_ok": False, "large_ok": False},
]
ROUTES = {"small": {"cost": 0.002, "latency": 300},
          "large": {"cost": 0.020, "latency": 900}}

def replay(threshold):
    records = []
    for case in CASES:
        route = "large" if case["difficulty"] >= threshold else "small"
        records.append({"id": case["id"], "route": route,
                        "ok": case[route + "_ok"], **ROUTES[route]})
    ordered = sorted(record["latency"] for record in records)
    return {"threshold": threshold,
            "success": mean(record["ok"] for record in records),
            "cost": sum(record["cost"] for record in records),
            "p95": ordered[ceil(0.95 * len(ordered)) - 1],
            "large": sum(record["route"] == "large" for record in records)}

def main():
    print("SYNTHETIC replay: 4 fixed tasks; invented route prices")
    print("threshold success total_cost p95_ms large_calls")
    for threshold in (0.3, 0.6, 0.9):
        result = replay(threshold)
        print(f"{result['threshold']:.1f} {result['success']:.2f} "
              f"{result['cost']:.3f} {result['p95']} {result['large']}")

if __name__ == "__main__":
    main()

Push it further

Add recorded redacted OpenTelemetry traces and compare critical-path attribution with the policy-level latency change.