Agentic Engineering Lab

Build it in stages

Write a task acceptance contract and a dataset datasheet with provenance and grouping rules.
Create at least 30 original cases across three meaningful slices, including unanswerable and tool-failure cases.
Add a deterministic evaluator and manually audit a sample of its decisions.
Run two fixed policies on identical task IDs and save a reproducibility manifest.
Compute paired counts, effect size, uncertainty, coverage, and cost with explicit missing-data rules.
Ablate one mechanism and write a decision memo that distinguishes evidence from untested assumptions.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

Every case has a stable ID, reference outcome, fixture version, and slice.
The agent cannot access hidden grading references during evaluation.
Paired results reconcile exactly to the reported totals.
The report states dataset limitations, resource budgets, and any repeated variant selection.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py

python

import json
from math import comb

RECORDS = [
    {"id": "a", "slice": "lookup", "old": 1, "new": 1},
    {"id": "b", "slice": "lookup", "old": 0, "new": 1},
    {"id": "c", "slice": "lookup", "old": 0, "new": 1},
    {"id": "d", "slice": "tool", "old": 1, "new": 0},
    {"id": "e", "slice": "tool", "old": 0, "new": 1},
    {"id": "f", "slice": "tool", "old": 1, "new": 1},
    {"id": "g", "slice": "boundary", "old": 1, "new": 1},
    {"id": "h", "slice": "boundary", "old": 0, "new": 0},
]

def compare(records):
    wins = sum(r["new"] > r["old"] for r in records)
    losses = sum(r["new"] < r["old"] for r in records)
    n = wins + losses
    p = 1.0 if not n else min(
        1.0, 2 * sum(comb(n, k) for k in range(min(wins, losses) + 1)) / 2 ** n)
    return {"tasks": len(records), "wins": wins, "losses": losses,
            "delta": (wins - losses) / len(records), "p_value": p}

def main():
    if len({r["id"] for r in RECORDS}) != len(RECORDS):
        raise ValueError("duplicate task ID")
    print("SYNTHETIC paired evaluation")
    print(json.dumps(compare(RECORDS), sort_keys=True))
    for group in sorted({r["slice"] for r in RECORDS}):
        rows = [r for r in RECORDS if r["slice"] == group]
        print(group, "old", sum(r["old"] for r in rows),
              "new", sum(r["new"] for r in rows), "n", len(rows))

if __name__ == "__main__":
    main()

Push it further

Add clustered bootstrap intervals and compare them with naive row bootstrap intervals on a dataset containing related document families.