We reconstruct the workflow from the user's intent to its final side effects. Every model call, tool action, queue message, database write, timer, approval, notification, and external dependency is placed on the map with its state and failure behavior. The key question is not only where an exception occurs, but what the system can safely do after it no longer knows whether an action completed.
AI Reliability Engineering
Stop losing agent state to retries, restarts, and human waits.
AI agent workflow development for long-running, failure-prone systems. Add durable state, safe retries, approvals, recovery tests, and operator visibility.
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.
System-level interventionProduction-shaped diagnosisInspectable verification
The production trigger
Recognize the failure before naming the service.
Make long-running agent work resumable, idempotent, and operable.
A tool call times out, a worker restarts, a human approval takes hours, or the model returns a different shape on retry. The workflow loses state because its history only existed inside a request and a growing context window.
Trigger detail 1We model the state explicitly, choose a framework that matches your operating constraints, migrate the highest-risk flow, and leave replay and recovery tests behind.
Trigger detail 2System diagnosis
Find where the failure actually lives.
The visible symptom is rarely the whole problem. The first pass follows it through the production path until the controllable boundary is clear.
Current incidents and operational workarounds reveal the durability requirements. We inspect lost conversations, duplicate actions, stuck jobs, manual database repairs, unbounded retries, missing cancellation, and flows that cannot explain their status to a user. We also identify state incorrectly kept in model context: business facts, progress, permissions, and tool results that should be explicit, queryable records rather than tokens replayed on every turn.
Framework selection follows the workflow's duration, throughput, consistency, infrastructure, language, deployment constraints, and operator experience. A short background task may not need the same orchestration as a multi-day approval process. This engagement fits a production flow with costly recovery or state loss. It does not justify replacing every queue and job merely to standardize on a fashionable runtime.
The intervention
Change the critical path, not the surrounding theatre.
Each workstream targets a different failure boundary. Together they connect product behavior, infrastructure, controls, and verification.
State and failure modeling
Define workflow states, transitions, invariants, side effects, timeouts, cancellation, approvals, and terminal outcomes. Each dependency receives a failure policy and each user-visible state receives a truthful status. We separate durable business state from ephemeral model context so operators can inspect progress without reconstructing a prompt transcript.
Activity and idempotency design
Wrap external effects in bounded activities with explicit inputs, outputs, timeouts, retry classes, and idempotency keys. We decide which errors are transient, permanent, or ambiguous and how duplicate execution is handled. Compensation is used where reversal is possible; where it is not, the workflow pauses for reconciliation instead of repeating a risky action.
Priority workflow migration
Move one high-value flow incrementally, preserving tested domain and model logic while extracting orchestration concerns. Existing APIs and events are adapted around a versioned workflow contract. Deployment accounts for in-flight work and old state, and feature controls allow measured exposure without forcing every customer onto an unproven path at once.
Recovery and operator experience
Instrument state transitions, attempts, waits, model and tool correlation, age, and terminal outcomes. Recovery tests terminate workers, delay callbacks, duplicate events, fail dependencies, and resume approvals. The runbook explains how to inspect, pause, retry, repair, cancel, and replay work without bypassing the workflow's invariants.
Inspectable changes
See what changes in the system.
- Workflow state and failure-mode map
- Framework decision record
- Durable workflow implementation
- Checkpoint and resume behavior
- Idempotent tool calls and retry policy
- Human-in-the-loop pause and escalation
- Recovery tests, observability, and operator runbook
Engineering judgment
The decisions that determine whether the change holds.
Durability boundary
Choose where a workflow begins and ends and which state belongs in the orchestrator. Making every function an activity creates noise; keeping important side effects inside opaque code defeats recovery. The boundary follows business outcomes, external effects, waiting periods, and the points where an operator or user needs durable visibility.
Retry and compensation
Define retry limits from operation semantics rather than a global backoff default. Read calls, model calls, payments, notifications, and record mutations have different duplicate risks. We specify idempotency, timeout responsibility, compensation, dead-letter or reconciliation paths, and the point at which automated recovery should stop and request human judgment.
History and model context
Decide what belongs in durable state, what becomes a summarized or retrieved model input, and what can expire. The model receives only the context required for its current decision, while the workflow retains authoritative progress and evidence. This limits token growth and prevents a regenerated summary from becoming the sole record of completed work.
Strong fit
- Your agent has state, long-running work, or human approval steps.
- Restarts and retries can duplicate work or lose context today.
Probably not the right fit
- Your agent has no state or long-running work.
- You require one specific orchestrator before assessing fit.
Scope, timing, and ownership
Know the commercial shape before the call.
A bounded engagement with a service-specific delivery sequence, an agreed price, and artifacts that stay under your control.
Days 1-3
Workflow and failure map
Identify state, side effects, retries, approvals, timeouts, and audit requirements.
Days 4-7
Durable model
Design activities, checkpoints, idempotency, compensation, and human-in-the-loop states.
Days 8-12
Migration
Move the priority workflow and connect observability, queues, and existing services.
Days 13-15
Recovery testing
Exercise restarts, duplicate events, delayed approvals, partial failures, and replay.
Typical fixed scope
Mid-four to low-five figures
Most engagements fall in this planning range. Your exact fixed price is agreed in writing after we review the production surface, access needs, and success criteria. The range is guidance, not a quote.
Commercial terms
No open-ended consulting meter.
- Fixed scope agreed in writing
- Fixed timeline agreed in writing
- Fixed price agreed before work starts
- Client-owned deliverables from day one
- No hourly meter or surprise overages
Client ownership
The implementation remains yours.
- Your repository
- Your infrastructure
- Your evals and evidence
- Yours from commit one
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.
Durable Workflows
Questions worth settling before the call
Technical, commercial, and handoff questions answered before you book.
Give the workflow a state model and operators a recovery path.
Bring the failing workflow, current evidence, and the constraints your engineers cannot ignore.
Book a 30-minute consultation
30 minutes. No deck. Leave with a clear next step.