AccessBridge
A verifiable assistant for navigating synthetic forms with accessible state and explicit confirmation
The problem worth solving
A form assistant can enter plausible values into the wrong field, overlook a validation message, lose keyboard focus, or report success before submission actually completes. These failures become particularly frustrating when a user relies on accessible names, keyboard interaction, or assistive technology. Build a local workflow assistant for fictional event registration and membership forms. It should explain the next action, resolve fields semantically, preview changes, preserve meaningful focus, and verify the resulting state. All records and submissions remain synthetic throughout the benchmark.
What could make it stand out
WebArena already measures browser task completion, and W3C guidance already defines accessibility expectations. The proposed contribution is a task contract combining functional correctness with observable accessibility state at each action boundary. A completed form counts as successful only when intended values, validation status, confirmation state, and focus behavior all match the contract. The project tests whether semantic binding and postcondition checks improve reliability under layout and label perturbations. It supports accessible workflows but does not certify WCAG conformance or replace evaluation with people who use assistive technology.
On held-out synthetic forms with reordered controls, duplicate labels, asynchronous validation, and modal steps, a semantic-action agent with preview and postcondition verification will reduce silent task failures by at least 40% compared with a browser agent that checks only the final visible page. Clean-form task success must remain within five percentage points of the baseline. Reject the hypothesis if the advantage disappears when both agents receive the same accessibility tree and action budget.
System architecture
| Component | Responsibility |
|---|---|
| Synthetic form suite | Create local forms for club membership, workshop registration, and fictional contact updates. Each form exposes a ground-truth state API used only by the evaluator and includes controlled accessibility and workflow perturbations. |
| Semantic observation adapter | Read roles, accessible names, values, enabled state, error associations, live-region updates, and focus. Keep screenshot evidence available for multimodal comparison while retaining a typed semantic representation. |
| Bounded workflow planner | Translate the user's fictional task into field-level intents and prerequisite steps. If multiple controls satisfy an intent, request clarification or stop rather than guessing from position. |
| Preview and action verifier | Show intended field changes and the current form revision before a synthetic submit action. Bind confirmation to the preview revision and recheck preconditions if dynamic validation changes the form. |
| Accessible interaction recorder | Record keyboard actions, focus transitions, validation announcements, and verified postconditions. Produce an explanation in simple language and retain a replay artifact that can be inspected alongside evaluator state. |
Data and reproducibility
Build 24 original local forms with five task variants each, yielding 120 tasks. Divide templates into development and held-out layouts. Include labeled, unlabeled, ambiguous, dynamic, and modal cases, but clearly mark intentionally inaccessible fixtures as benchmark faults. W3C documents inform expected behaviors; WebArena informs reproducible browser-task evaluation rather than providing accessibility certification.
Start with a synthetic benchmark
Generate fictional names and non-sensitive values, then perturb control order, label wording, validation timing, required status, and modal focus placement. Keep the requested outcome constant across paired variants. Deliberately add two controls with the same accessible label to test safe ambiguity handling. Introduce a form revision between preview and confirmation to check that stale approval cannot commit unintended changes. Hold out complete layouts to prevent memorized selectors from dominating.
Baselines you must beat
- A selector-based scripted workflow using stable fixture IDs, serving as an upper-bound diagnostic for intended form behavior rather than a general agent.
- A bounded browser agent that reads the same accessibility tree but verifies only the final page text.
- A semantic-action agent without revision-bound previews or intermediate postcondition checks, isolating the contribution of verification.
Measure the claim
Targets below are proposed success criteria. No results have been achieved on your behalf.
| Metric | Definition and target |
|---|---|
| Silent failure rate | Fraction of tasks the assistant declares complete when evaluator state shows an incorrect value, absent submission, unresolved error, or failed required focus transition. Target, not achieved result: at least 40% relative reduction versus final-page-only verification on held-out perturbed forms. |
| Verified task success | Fraction of tasks meeting both functional state and the declared accessibility-state contract, with ambiguous tasks scored for correct clarification or abstention separately. Target, not achieved result: at least 85% on clean held-out forms and no more than five percentage points below the comparable baseline. |
| Unintended commitment and user effort | Count synthetic submissions without a current matching preview confirmation, plus extra user clarification turns and keyboard actions per successful task. Target, not achieved result: zero stale-preview commitments in deterministic fixtures; report median clarification and action overhead without hiding failures. |
Experiments and ablations
- Pair each clean form with reordered controls, renamed labels, or dynamic validation while preserving the target outcome. Compare typed semantic selection with positional or memorized selector strategies.
- Trigger a revision change after preview, then attempt confirmation. Verify that the assistant refreshes the preview and never describes an uncommitted action as completed.
- Compare accessibility-tree-only and multimodal observation under identical budgets. Where practical, invite a small compensated review by assistive-technology users; report such feedback as qualitative and do not invent a user study if none occurs.
Your execution plan
0 of 12 deliverables checked. Tick a deliverable after recording evidence in your project repository.
Define task and accessibility contracts
Build the local form benchmark
Run browser baselines
Add preview and postconditions
Evaluate perturbation robustness
Package the workflow study
Working seed
Run this small, deterministic core first. It demonstrates the central mechanism. The complete system, experiments, and deployment are your capstone work.
"""AccessBridge: bind semantic fields and verify a synthetic form commit."""
from copy import deepcopy
import hashlib
import json
def fingerprint(form):
content = json.dumps(form, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(content.encode()).hexdigest()
def preview(form, requests):
operations = []
for label, value in requests.items():
matches = [field for field in form["fields"]
if field["label"] == label and field["role"] == "textbox"
and not field["disabled"]]
if len(matches) != 1:
return {"status": "abstain", "reason": "ambiguous_or_missing_field"}
operations.append({"id": matches[0]["id"], "label": label, "value": value})
return {"status": "preview", "revision_hash": fingerprint(form), "operations": operations}
def commit(form, plan, confirmation):
if plan["status"] != "preview" or confirmation != plan.get("revision_hash"):
return {"status": "rejected", "reason": "confirmation_mismatch"}
if fingerprint(form) != plan["revision_hash"]:
return {"status": "rejected", "reason": "stale_preview"}
working = deepcopy(form)
for operation in plan["operations"]:
field = next(f for f in working["fields"] if f["id"] == operation["id"])
field["value"] = operation["value"]
if any(f["required"] and not f["value"].strip() for f in working["fields"]):
return {"status": "rejected", "reason": "required_field_empty"}
working["revision"] += 1
working["status_message"] = "Synthetic form submitted"
working["focus"] = "receipt"
form.clear()
form.update(working)
return {"status": "verified", "field_ids": [o["id"] for o in plan["operations"]],
"status_message": form["status_message"], "focus": form["focus"]}
def main():
form = {"revision": 1, "focus": "name", "status_message": "", "fields": [
{"id": "name", "label": "Applicant name", "role": "textbox", "value": "",
"disabled": False, "required": True},
{"id": "city", "label": "City", "role": "textbox", "value": "",
"disabled": False, "required": True}]}
request = {"Applicant name": "Sample Applicant", "City": "Pune"}
plan = preview(form, request)
stale_form = deepcopy(form)
stale_form["revision"] += 1
stale = commit(stale_form, plan, plan["revision_hash"])
assert stale["reason"] == "stale_preview"
ambiguous = deepcopy(form)
ambiguous["fields"].append(dict(form["fields"][0], id="second_name"))
assert preview(ambiguous, request)["status"] == "abstain"
result = commit(form, plan, plan["revision_hash"])
assert result["status"] == "verified" and form["focus"] == "receipt"
assert form["fields"][0]["value"] == "Sample Applicant"
print(json.dumps({"stale_attempt": stale, "successful_commit": result}, sort_keys=True, indent=2))
if __name__ == "__main__":
main()
Failure modes to investigate
- Correct DOM values do not establish that a workflow is usable with assistive technology. Verify focus and announcements and describe any missing real-user evaluation as a study limitation.
- An assistant could appear capable by relying on fixture-only IDs. Hold out layouts and give the general agents semantic observations rather than evaluator labels.
- Dynamic forms can invalidate a previously reviewed action. Bind synthetic commitment to an exact preview revision and require a refreshed preview after relevant state changes.
Your demo, moment by moment
- Ask the assistant to complete a fictional workshop form and watch it explain the intended values through accessible labels.
- Introduce two identically named fields. Show a clarification or abstention instead of a positional guess, then disambiguate the synthetic task.
- Display the change preview, modify a required field behind the scenes, and demonstrate rejection of the stale confirmation.
- Submit the refreshed synthetic form, verify its receipt and focus target, and compare silent-failure results across held-out layouts.
Write the resume bullet after the experiment
Replace every placeholder with your actual measurements. Keep the dataset size, baseline, and evaluation conditions available for interview questions.
A research extension
Add a second local interaction mode such as spoken instructions with text confirmation, while retaining the same task-state verifier. Study whether shorter explanations reduce user effort without increasing ambiguity. A more rigorous future study can recruit participants with relevant access needs and establish an appropriate research protocol; simulated persona labels alone must never be presented as evidence of accessibility for those users.
Related work to challenge your idea
Defines accessibility criteria including programmatically available component information and status messages. The project selects observable workflow checks and must not describe them as complete conformance testing.
Supplies established keyboard and focus guidance for dialogs. These behaviors inform synthetic task contracts and expected postconditions.
Provides realistic reproducible web-agent tasks and functional evaluation. This proposal focuses on a smaller form suite with explicit accessibility-state and confirmation checks.