Durable jobs, acknowledgements, and stale workers
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Design job submission around durable state and duplicate delivery.
- Place checkpoints around recoverable workflow transitions.
- Use leases and fencing tokens to reject stale ownership.
The HTTP request is not the job lifetime
A useful agent task may take minutes and outlive the connection that submitted it. Treat submission as creating a durable job with a stable identifier, then let the client inspect status or receive progress through a separate channel. The API validates the request and authorization, records accepted work, and returns the identifier. A disconnected browser should not erase the job's identity or cause a duplicate submission to launch unrelated work.
Define explicit states such as queued, running, waiting for approval, succeeded, failed, and cancelled. Record the state version and the last confirmed step. Keep large artifacts outside small queue messages, referencing immutable versions or controlled storage locations. A message should carry enough information to find the authoritative job record, not become the only copy of all state required to recover the workflow.
There are two acknowledgement boundaries
A queue publisher needs evidence that the broker accepted its message. A consumer acknowledges after handling a delivery according to the application's durability contract. RabbitMQ documents publisher confirms and consumer acknowledgements as distinct mechanisms. Neither by itself proves that a remote business action happened exactly once. A worker can perform an action and crash before acknowledging, causing the same message to arrive again.
Therefore, design consumers for duplicate delivery. Use stable operation identifiers and durable result records, and reconcile ambiguous external outcomes. If a database update and queue publication must agree, an outbox can record the intended message in the same local transaction as the state change, with a separate dispatcher sending it. The dispatcher can publish more than once, so the receiving side still needs idempotent handling. Moving the boundary does not remove the duplicate problem.
Checkpoint decisions, not only conversation text
A checkpoint should contain enough state to resume without repeating irreversible work: workflow version, completed step identifiers, tool receipts, remaining budgets, approval references, and relevant artifacts. A conversation transcript can help explain how the agent arrived at a decision, but it is not a substitute for a machine-readable state transition. Keep sensitive material subject to the same access and retention controls as other application data.
In a toy workflow, fetch, summarize, and publish are three steps. After summarization, store the exact summary artifact and its version before moving toward publication. On restart, the worker can reuse that artifact and check whether publication was already confirmed. Re-running summarization might generate different content and invalidate an existing approval. Recovery should resume a known plan version or explicitly migrate it, not silently reinterpret the task under a changed prompt.
A lease does not stop the old worker
A lease gives a worker ownership until a deadline. If worker A pauses and its lease expires, worker B can claim the job. A may later resume, unaware that it lost ownership. Checking expiry only in A's memory is insufficient. Assign a monotonically increasing fencing token at each successful claim, and require updates to match the current token at the authoritative state boundary.
For example, A claims token 4 until time ten. B claims token 5 at time eleven. A's completion using token 4 must be rejected even if its computation was correct. The lab demonstrates this state check. It does not prevent A from calling an external service that ignores the token. Consequential resources need their own idempotency or fencing contract, or operations must be mediated through a component that can enforce the current ownership.
Model the failure before adding infrastructure
The accompanying example simulates a durable checkpoint with a Python dictionary so the transition order is visible. A crash is an exception injected after a confirmed step; restarting calls the function again with the same dictionary. This does not survive process termination and is not a queue or workflow engine. It illustrates which data a real store would need to preserve.
For a production design, draw the crash windows around each database commit, queue acknowledgement, and remote action. Decide what recovery observes in each window. Test duplicate messages, expired leases, old tokens, and changed workflow versions. A reliable worker should either advance a valid state transition or leave an explicit recoverable outcome. An unbounded loop that repeatedly asks the model what to do next is not a replacement for a documented recovery protocol.
Work through the code
A deliberate exception simulates a stop after the checkpoint has been updated. Resuming skips completed synthetic steps. The dictionary is not durable, and the example omits the harder crash between a real side effect and its checkpoint. Extend it by modeling that ambiguity with operation receipts.
checkpoint = {"completed": [], "artifacts": {}}
effects = []
def run(state, crash_after=None):
for step in ("fetch", "summarize", "publish"):
if step in state["completed"]:
continue
artifact = {"fetch": "source-v8", "summarize": "summary-v1",
"publish": "receipt-17"}[step]
effects.append(step)
state["artifacts"][step] = artifact
state["completed"].append(step)
if step == crash_after:
raise RuntimeError("simulated worker stop")
return state["artifacts"]["publish"]
try:
run(checkpoint, crash_after="summarize")
except RuntimeError:
print("checkpoint:", checkpoint["completed"])
print("receipt:", run(checkpoint))
print("effects:", effects)
checkpoint: ['fetch', 'summarize'] receipt: receipt-17 effects: ['fetch', 'summarize', 'publish']
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
Worker A finishes an external upload and crashes before its queue acknowledgement. The message is delivered to B. Why should B not simply upload again, and which identifiers should it inspect?
Check your understanding
A stale worker has token 4 while the authoritative job now has token 5. Which control actually rejects its completion?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.