Workspace/Mini projects
Loading progress
All mini projects
MODULE 28 · 6 HOUR BUILD

Reward shortcut audit bench

Build a small evaluation bench that exposes candidates which score highly under a proxy reward while failing an independent task criterion.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Run the seed and inspect the difference between substring reward and exact submitted-answer correctness.
  2. Add at least 30 authorized synthetic cases spanning correct, partial, malformed, and adversarial outputs.
  3. Version two candidate rewards and preserve a separate independent outcome label.
  4. Report agreement, false acceptance, and selected-candidate success by failure category.
  5. Document which reward defects were fixed and which outcomes the verifier still cannot assess.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • All fixtures are synthetic or authorized, with an explicit expected outcome.
  • At least five candidates deliberately obtain proxy reward without meeting the task contract.
  • Metrics state denominators and distinguish candidate-level checks from selected-answer success.
  • The report includes a regression where a stricter reward rejects a previously valid candidate, or shows that the tested set contains none.
  • No output is described as measured model-training improvement because the seed does not train a model.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
import json

CASES = [
    {"id": "a", "answer": "42", "candidates": ["41 42 43", "42"]},
    {"id": "b", "answer": "7", "candidates": ["I considered 7; final answer: 8", "7"]},
    {"id": "c", "answer": "12", "candidates": ["312", "12"]},
    {"id": "d", "answer": "5", "candidates": ["5", "unknown"]},
]

def proxy(candidate, answer):
    return float(answer in candidate)

def strict(candidate, answer):
    return float(candidate.strip() == answer)

def audit(cases):
    report = []
    for case in cases:
        candidates = case["candidates"]
        chosen = max(range(len(candidates)),
                     key=lambda i: (proxy(candidates[i], case["answer"]), -i))
        strict_chosen = max(range(len(candidates)),
                            key=lambda i: (strict(candidates[i], case["answer"]), -i))
        report.append({
            "id": case["id"],
            "proxy_selected": candidates[chosen],
            "proxy_selected_correct": bool(strict(candidates[chosen], case["answer"])),
            "strict_selected_correct": bool(strict(candidates[strict_chosen], case["answer"])),
            "shortcut_count": sum(proxy(c, case["answer"]) > strict(c, case["answer"])
                                  for c in candidates),
        })
    return report

if __name__ == "__main__":
    rows = audit(CASES)
    result = {"fixture": "synthetic reward audit", "cases": rows,
              "proxy_selected_successes": sum(r["proxy_selected_correct"] for r in rows),
              "strict_selected_successes": sum(r["strict_selected_correct"] for r in rows),
              "denominator": len(rows)}
    assert result["proxy_selected_successes"] == 1
    assert result["strict_selected_successes"] == 4
    print(json.dumps(result, indent=2, sort_keys=True))

Push it further

Add a simulated policy that changes its candidate-selection probabilities under reward pressure, then test whether independent success rises or falls as optimization continues.