Workspace/Mini projects
Loading progress
All mini projects
MODULE 27 · 6 HOUR BUILD

Adaptation readiness report

Create a preflight report that combines dataset lineage, retained task coverage, protected-group checks, and adapter memory estimates before a training job is approved.

Build evidence Record your actual checks, results, and limitations.

Build it in stages

  1. Extend the seed with validated JSONL ingestion and source/group identifiers.
  2. Produce exclusion reasons and retained counts by task type, including missing categories.
  3. Add configurable base precision, adapter rank, optimizer bytes, and a separately labeled activation allowance.
  4. Create a protected evaluation manifest and inspect a sample of near-duplicate candidates.
  5. Write a short experiment proposal comparing a frozen baseline, prompt revision, and adapter run.

Your acceptance criteria

Use these as your project review. Record commands, outputs, and failure cases in your repository.

  • The same fixture and configuration produce identical report content.
  • Every input record is retained or has a recorded exclusion reason.
  • No protected group appears in retained training records.
  • Parameter and weight-byte estimates pass hand-checked small examples.
  • The report labels its memory estimate as incomplete until validated by a measured training run.

A working starting point

The seed runs as supplied. Extend it to satisfy the full brief. It is a teaching starting point, not a finished portfolio submission.

main.py
python
import json
import unicodedata
from collections import Counter

RECORDS = [
    {"id": "a", "group": "t1", "task": "recovery", "prompt": "Reset password"},
    {"id": "b", "group": "t1", "task": "recovery", "prompt": " RESET password "},
    {"id": "c", "group": "eval1", "task": "export", "prompt": "Export records"},
    {"id": "d", "group": "t2", "task": "escalate", "prompt": "Contact an operator"},
]

def canonical(text):
    return " ".join(unicodedata.normalize("NFKC", text).casefold().split())

def report(records, heldout):
    seen, retained, excluded = set(), [], []
    for record in records:
        key = canonical(record["prompt"])
        if record["group"] in heldout:
            excluded.append({"id": record["id"], "reason": "heldout"})
        elif key in seen:
            excluded.append({"id": record["id"], "reason": "duplicate prompt"})
        else:
            seen.add(key)
            retained.append(record)
    rank, width, projections = 8, 4096, 32
    adapter_parameters = rank * (width + width) * projections
    return {
        "input_rows": len(records),
        "retained_ids": [row["id"] for row in retained],
        "coverage": dict(sorted(Counter(row["task"] for row in retained).items())),
        "excluded": excluded,
        "adapter_parameters": adapter_parameters,
        "adapter_fp16_bytes": adapter_parameters * 2,
        "memory_scope": "adapter weights only; excludes base, optimizer, activations",
    }

if __name__ == "__main__":
    result = report(RECORDS, {"eval1"})
    assert result["retained_ids"] == ["a", "d"]
    assert result["adapter_parameters"] == 2097152
    print(json.dumps(result, indent=2, sort_keys=True))

Push it further

Implement approximate duplicate candidate generation with token shingles, then measure false merges and misses against a manually labeled pair set.