# Durable Execution for AI Agents: Surviving Hours-Long Runs

> A job queue retries an agent from step one and re-pays for every model call. Durable execution replays a journal instead, skipping the steps that finished.

URL: https://www.matthewswong.com/en/blog/durable-execution-ai-agent-workflows/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-09-08
- Updated: 2026-09-08
- Category: Backend

> **Key takeaway:** Durable execution records every completed step of a long-running AI agent to a journal, so a crash resumes at the failed step instead of restarting the run. A job queue retries the whole job and re-pays for every LLM call already made. Temporal, Restate, DBOS and Inngest all sell that one guarantee.

The escalation scheduler in hierarchical-approval exists because a purchase order can sit at level two for three days while the finance director is on leave. I gave the engine an injectable Clock and a ManualClock for tests precisely so a process measured in days could be verified in milliseconds. The same shape turns up the moment an agent runs long enough to fail partway. An agent that dies on its ninth tool call and is restarted from the first by an ordinary job queue does not merely lose time: it re-pays for the eight model calls that already succeeded. That is what a retry means once the work is metered by the token.

That second failure is not a queue bug. It is a granularity mismatch: a queue's unit of retry is the job, and the job was the entire agent run. Durable execution moves the unit of retry down to the step. This post is what I learned comparing four runtimes that do it, what the replay model demands of your code, and the cases where none of it is worth the operational cost.

## A retry restarts the job; a replay skips what already finished

The finding first: a job queue and a durable runtime both survive a crash, but they resume in different places. BullMQ's contract is that a failed job runs again, the whole handler, from its first line. Restate's contract is that the handler runs again and every step that already completed returns its recorded result instead of executing. Restate's own documentation puts it plainly: it replays the journal, skipping completed steps and resuming from exactly where it left off. What gets written to that journal is the whole trick.

- Step outputs. The text the model returned, the row the database wrote, the file the tool produced, serialised as JSON and replayed instead of recomputed.
- Timers. A sleep is a journal entry, so Restate's docs give the example of a handler sleeping twelve hours that fails after eight and then sleeps only four more, not twelve.
- External signals. An approval that arrived while your process was down is stored durably until the invocation is ready to read it, rather than being dropped on the floor.

That is why the difference is a money question rather than a taste question. For a CRUD job, re-running from the top costs a few hundred milliseconds. For an agent that has already made eight model calls, fetched three PDFs and written a row, re-running from the top costs the eight calls again, plus a duplicate row for any step that was not idempotent.

## The same agent loop, billed twice and billed once

Here is the shape of the mistake. The naive version is one queue job wrapping the whole loop, and it is the version almost everyone writes first, because it is the version that works on the happy path and reads perfectly well in review.

The durable version is not more code, it is differently bounded code. Each paid call sits inside its own ctx.run. On the second attempt the handler still runs from the top, that part is unchanged, but the plan step and the seven page summaries return from the journal without touching the model, and execution genuinely continues at page eight. Inngest describes the identical mechanism in different words: the function is re-executed with the state of the previous execution, and for a step that already succeeded the SDK injects the result into the return value instead of running the code.

## Determinism is the price of replay, and an LLM cannot pay it

Replay only works if the code around the steps makes the same decisions in the same order every time. Temporal states the constraint directly: any time your workflow code is executed it must make the same API calls in the same sequence given the same input, and non-deterministic operations such as API calls, LLM invocations and database queries belong in Activities. People bounce off this rule because it looks arbitrary until you internalise the asymmetry: the orchestration path is executed many times, the steps are executed once.

Notice what that makes a model call. An LLM can return a different answer to the same prompt, so it can never live in the deterministic path; it is a step by construction. That is convenient rather than restrictive. The thing you most want to avoid re-running is exactly the thing the runtime forces you to isolate, which means a correctly written durable agent gets cost recovery as a side effect of satisfying the determinism rule.

> Use the runtime's seeded random for idempotency keys, not the platform UUID function. Restate's ctx.rand.uuidv4 is seeded by the invocation ID, so a retry sends the vendor the same key it saw the first time. Generate that key with crypto.randomUUID instead and every replay looks like a brand-new request to the payment or model API you were trying to deduplicate against.

## Four runtimes, and where each one keeps your state

The interesting difference between these products is not the API. They converge on the same idea of wrapping the side effect. The difference is what you have to run and where the journal lives, and that is what decides whether adopting one is an afternoon or a quarter.

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| Temporal | Event History, held by the Temporal Service | A Temporal Service plus your own Worker processes | A Worker replays the code and matches each Command against Events already in the history | Go, Java, TypeScript, Python, .NET, PHP, Ruby |
| Restate | A journal plus an embedded key-value store inside the Restate Server | The Restate Server; your handlers stay ordinary HTTP services | Restate re-invokes the handler and replays the journal, skipping completed steps | TypeScript, Java, Kotlin, Go, Python, Rust |
| DBOS | Checkpoints in a Postgres system database you already own | Nothing beyond Postgres; the library runs in your process | The app finds workflows left pending and re-runs them, returning checkpointed step outputs | Python, TypeScript, Go, Java |
| Inngest | Memoised step state in Inngest's managed function state store | An HTTP endpoint served by your app; Inngest orchestrates from its side | Your function is re-invoked with previous state and the SDK injects memoised step results | TypeScript, Python, Go |

DBOS is the outlier worth noticing if you already run Postgres. Its architecture page is explicit that there is no separate orchestration server and no infrastructure required besides Postgres, because the library checkpoints workflows and steps into a system database inside your own cluster. Temporal is the opposite trade: a service to operate, in exchange for the deepest story on visibility, versioning and polyglot workers. Restate sits between them, with the server as the only new process and handlers that remain plain HTTP services.

## Waiting three days for a human is the same primitive as waiting for a timer

This is where the ERP work and the agent work turned out to be one problem. In hierarchical-approval an instance legitimately sits pending until a person acts, slaDeadlineDays defines when the escalation scheduler should care, and the engine accepts a Clock so those days can be simulated. A test advances a ManualClock by three days, ticks the scheduler, and the SLA breach event fires with zero wall-clock time elapsed. I built that because there was no other way to test a three-day deadline in CI.

A durable runtime gives you those two primitives as language-level constructs instead of as something you build. A durable sleep is a journal entry the server owns, so it outlives the process that started it. An awakeable hands an external system a one-shot token to resolve, and Restate's own table lists agent steering and human approvals as precisely what signals are for. An agent pausing for sign-off and an approval pausing for a director are the same shape; I had written a narrow, domain-specific version of the general thing without knowing there was a general thing.

> hierarchical-approval is my own npm package, a TypeScript-first hierarchical approval engine for ERP developers, MIT licensed and currently at 4.0.0. It is not built on a durable execution runtime: it persists through its own storage adapters and runs its own escalation scheduler. That is the honest comparison, the same requirements solved narrowly for one domain rather than generally.

## The costs the journal quietly adds

Three of these surprised me, and all three are documented by the vendors themselves rather than discovered in anyone's incident review. None of them is a reason to avoid durable execution, but each one is a reason to read the runtime's docs before the agent is in production rather than after.

- Long steps need the timeout raised. Restate suspends an invocation and retries it if it receives no new journal entry for more than a minute by default, and its docs call out an LLM call taking up to three minutes as the case where you must adjust the service's inactivity and abort timeouts.
- Everything a step returns is serialised into the journal. Returning a whole scraped page from a tool step writes that page into the log; returning the three extracted fields does not. I now treat a step's return type as a storage decision, because it is one.
- Deploying became a versioning problem. Temporal names non-deterministic code changes, shipping new workflow code while executions started on the old code are still running, as one of the two main sources of non-determinism, and Restate warns that a long sleep means keeping the deployment version alive until the invocation completes.

The journal is a real artefact with a size, a schema and a compatibility surface, and it now sits between your deploy pipeline and your running agents. Treat it the way you treat a database migration, not the way you treat a log file.

> The dangerous failure is not a crash, it is a wrong resume. Ship a changed orchestration path while runs are in flight and a replay can reach a decision point that no longer exists in the code, so the run either trips a non-determinism check or silently takes a branch nobody intended. Version workflows deliberately and keep old versions serving until their runs drain; orchestration code is not application code you can hotfix.

## When you do not need any of this

The unfashionable part, stated plainly: most agent calls should never go near a workflow engine. A single tool call that returns in five seconds has nothing to resume. If it fails, run it again and you have lost five seconds and one model call. Durable execution earns its cost only when a run is long enough to be interrupted, expensive enough that repeating it hurts, or paused on something outside your own process. Here is the ladder I now climb, and I stop at the first rung that holds.

1. An idempotency key and a steps table in Postgres. Record each completed step with its output, and skip the ones already recorded. This is durable execution's core idea with none of its infrastructure, and it is enough for a three-step pipeline.
2. An in-process library. DBOS runs inside your application and needs nothing but the Postgres you already have, which makes it the smallest genuine step up from hand-rolling the steps table.
3. A hosted orchestrator. Inngest keeps the state on its side and asks only that your app expose an HTTP endpoint, so nothing new joins your estate to be patched, backed up and monitored.
4. A full workflow platform. Temporal, when you have many long-lived workflows, more than one language, and somebody whose job description includes operating it.

For a two-person shop, that fourth rung is a real bill paid in attention: a service to run, a versioning discipline to hold, and a new class of failure to learn to read. I would not pay it for one agent. I would pay it for a fleet, and the honest signal that you have a fleet is when two different teams have independently written their own steps table.

The rule I carry out of this is about granularity, not about products. Ask what your unit of retry is. If the answer is the whole job, every failure charges you again for the work that already succeeded, and with agents that work is metered by the token. Make the step the unit, with a journal, a steps table, or whatever matches the size of the problem, and a crash stops being an invoice.

## Sources and further reading

- Temporal - Workflow Definition: deterministic constraints and what belongs in an Activity
- Restate - Durable Steps: ctx.run journaling, deterministic time and seeded randoms
- Restate - Signals and external events: awakeables, signals and human approvals
- DBOS - Architecture: in-process library, Postgres system database and workflow recovery
- Inngest - How functions are executed: step memoisation and re-invocation

*Read the bottom line of each panel. The three differ only in what they treat as the unit of retry, and that choice is what decides whether a crash sends you a second invoice.*

*The middle panel is the one to get right first. Anything that bills you belongs in a durable step, because that is the only way a replay can skip it instead of paying for it twice.*

## Source links

- https://docs.temporal.io/workflow-definition
- https://docs.restate.dev/develop/ts/durable-steps
- https://docs.restate.dev/develop/ts/external-events
- https://docs.dbos.dev/architecture
- https://www.inngest.com/docs/learn/how-functions-are-executed

## FAQ

### What is durable execution, and how is it different from a job queue retry?

Durable execution writes every completed step of a run to a journal, so a restart replays that journal and skips the work that already finished. A job queue's unit of retry is the whole job, so a failure on step nine re-runs steps one to eight from scratch. The difference is the granularity of recovery, not the reliability of the storage.

### Why does retrying a failed AI agent run cost real money?

Every LLM call and paid tool call in an agent run is billed when it happens, not when the run finishes. If the run crashes at step nine and the retry restarts at step one, you pay for the eight calls that already succeeded a second time. Durable execution avoids that by returning the recorded output of each completed step instead of calling the model again.

### Do I need Temporal to build a long-running AI agent?

No. A single agent call that returns in a few seconds has nothing worth resuming, and an idempotency key plus a steps table in Postgres is enough for a short pipeline. Reach for a workflow platform when runs are long enough to be interrupted, expensive enough that repeating them hurts, or paused waiting on something outside your process. Temporal is a service you have to operate, which is a real cost for a small team.

### Why must an LLM call live in an activity or step, not in workflow code?

Workflow code has to be deterministic because the runtime executes it repeatedly during replay, and Temporal's docs require that it makes the same API calls in the same sequence given the same input. A model can return a different answer to the same prompt, so it can never satisfy that rule. Putting the call in an activity or a ctx.run step means the result is journalled once and replayed thereafter.

### How can a workflow wait days for a human approval without holding a process open?

Durable runtimes persist the wait itself. Restate's durable sleep is a journal entry owned by the server, so a handler that slept eight of twelve hours before crashing sleeps only the remaining four. For a human decision you use an awakeable or a named signal: the runtime suspends the invocation, and an external system resolves the token whenever the person actually acts.

---

- HTML version: https://www.matthewswong.com/en/blog/durable-execution-ai-agent-workflows/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/durable-execution-ai-agent-workflows/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json