Deadlines, retry ownership, and ambiguous success

A tool call has at least two histories: what the caller observed and what the service actually did. Suppose an agent asks an inventory service to reserve one item. The service commits the reservation, but the reply disappears. The caller sees a timeout. It does not know whether the reservation exists. Retrying a read is usually straightforward; retrying a mutation can reserve another item. A reliable controller therefore records an operation identifier before attempting the mutation and uses the same identifier when reconciling or retrying it.

Classify failures at the boundary that understands the operation. A temporary connection failure may be retryable. A malformed account identifier is usually permanent until the request changes. A permission rejection needs a policy decision, not repeated requests. A syntactically valid tool response can still represent failure. Preserve the service error category separately from the exception raised by the client library.

A deadline is the time by which the entire user operation must finish. A timeout is a limit applied to one wait or attempt. If a request starts at monotonic time 10.0 with a 2.0 second allowance, its deadline is 12.0. A first attempt taking 0.4 seconds and a backoff of 0.3 seconds leave 1.3 seconds. The next attempt cannot receive a fresh 2.0 second timeout without breaking the original promise. Reserve some of the remaining time for parsing, validation, and returning a useful failure.

Use a monotonic clock for elapsed time because wall clocks can be adjusted. Pass the remaining budget down the call graph instead of independently inventing timeouts in every helper. A cancellation request can stop local waiting without proving that a remote mutation stopped. Reconciliation remains necessary for operations whose outcome is uncertain.

Consider 100 workers receiving an overload response simultaneously. Retrying every request exactly one second later creates another burst of 100 arrivals. Exponential backoff spaces repeated attempts farther apart; jitter spreads different callers within a window. A common design samples uniformly between zero and a capped exponential delay. The cap prevents a single wait from growing indefinitely. Jitter reduces synchronized retries, but it does not create downstream capacity.

Retry ownership matters as much as the formula. If an outer workflow permits three attempts and each of three nested layers also permits three, one original request can generate many leaf calls. A controller should choose where retries occur, expose attempt counts, and bound total work. Respect a service's documented retry guidance while ensuring that its suggested delay still fits the caller's deadline. If it does not fit, return a bounded failure.

An idempotency key is useful only when the receiver binds it to the operation and remembers the result. Reusing key order-17 with a different quantity must produce a conflict, not silently return a receipt for the old quantity. Scope the key by tenant and action, and retain it long enough to cover the realistic retry and recovery interval. Generating a fresh key on every attempt defeats duplicate suppression.

In an original toy calculation, a service succeeds independently with probability 0.8 per attempt. Up to three attempts yield success probability 1 minus 0.2 cubed, or 0.992, and expected attempts 1 plus 0.2 plus 0.04, or 1.24. Those numbers depend on independence. During an outage, failures are correlated and retries mainly add load. Measure actual recovery by error class instead of treating the independence calculation as a production guarantee.

The accompanying simulator represents time with numbers and never sleeps or contacts a service. That makes the deadline decisions easy to examine. A real client must additionally implement cancellation, connection management, observability, concurrency limits, and the receiver's duplicate suppression contract. Keep these distinctions explicit when explaining a portfolio project.

Record the operation identifier, attempt number, remaining budget, classified outcome, and whether the result was reconciled. Avoid logging credentials or entire sensitive requests. A failure response should distinguish exhausted attempts from exhausted time because they suggest different fixes. If an operation has no safe retry contract, the appropriate recovery may be a status lookup or a human decision. Reliability includes stopping with an accurate account of uncertainty, not merely increasing the number of attempts until something returns.

The durations describe predetermined failed attempts followed by success on the final attempt. Time is simulated, and backoff is deterministic so the output can be checked. Lower the deadline to see an attempt suppressed; this is a policy illustration, not a network retry client.