Durable Execution for AI Agents: Surviving Hours-Long Runs

The escalation scheduler in hierarchical-approval exists because a purchase order can sit at level two for three days while the finance director is on leave. I gave the engine an injectable Clock and a ManualClock for tests precisely so a process measured in days could be verified in milliseconds. The same shape turns up the moment an agent runs long enough to fail partway. An agent that dies on its ninth tool call and is restarted from the first by an ordinary job queue does not merely lose time: it re-pays for the eight model calls that already succeeded. That is what a retry means once the work is metered by the token.

That second failure is not a queue bug. It is a granularity mismatch: a queue's unit of retry is the job, and the job was the entire agent run. Durable execution moves the unit of retry down to the step. This post is what I learned comparing four runtimes that do it, what the replay model demands of your code, and the cases where none of it is worth the operational cost.

A retry restarts the job; a replay skips what already finished

The finding first: a job queue and a durable runtime both survive a crash, but they resume in different places. BullMQ's contract is that a failed job runs again, the whole handler, from its first line. Restate's contract is that the handler runs again and every step that already completed returns its recorded result instead of executing. Restate's own documentation puts it plainly: it replays the journal, skipping completed steps and resuming from exactly where it left off. What gets written to that journal is the whole trick.

• Step outputs. The text the model returned, the row the database wrote, the file the tool produced, serialised as JSON and replayed instead of recomputed.
• Timers. A sleep is a journal entry, so Restate's docs give the example of a handler sleeping twelve hours that fails after eight and then sleeps only four more, not twelve.
• External signals. An approval that arrived while your process was down is stored durably until the invocation is ready to read it, rather than being dropped on the floor.

That is why the difference is a money question rather than a taste question. For a CRUD job, re-running from the top costs a few hundred milliseconds. For an agent that has already made eight model calls, fetched three PDFs and written a row, re-running from the top costs the eight calls again, plus a duplicate row for any step that was not idempotent.

The same agent loop, billed twice and billed once

Here is the shape of the mistake. The naive version is one queue job wrapping the whole loop, and it is the version almost everyone writes first, because it is the version that works on the happy path and reads perfectly well in review.

// Wrong: one BullMQ job for the entire agent run.
// A throw on the ninth tool call re-queues the job, and attempt 2 restarts
// at line 1 — the eight model calls before the crash are billed again.
worker.process("tender-review", async (job) => {
  const plan = await llm.plan(job.data.brief);      // paid
  const notes = [];
  for (const url of plan.sources) {
    notes.push(await summarisePage(url));           // paid, once per source
  }
  return llm.compose(notes);                        // paid
});

// Right: every paid call is its own journalled step.
// Attempt 2 still re-enters the handler from the top, but each completed
// ctx.run returns its recorded result instead of calling the model again.
export const tenderReview = restate.workflow({
  name: "TenderReview",
  handlers: {
    run: async (ctx: restate.WorkflowContext, brief: string) => {
      const plan = await ctx.run("plan", () => llm.plan(brief));

      const notes = [];
      for (const url of plan.sources) {
        // One journal entry per source, so a resume lands on the source
        // that actually failed rather than on the first one.
        notes.push(await ctx.run("page:" + url, () => summarisePage(url)));
      }

      return ctx.run("compose", () => llm.compose(notes));
    },
  },
});

The durable version is not more code, it is differently bounded code. Each paid call sits inside its own ctx.run. On the second attempt the handler still runs from the top, that part is unchanged, but the plan step and the seven page summaries return from the journal without touching the model, and execution genuinely continues at page eight. Inngest describes the identical mechanism in different words: the function is re-executed with the state of the previous execution, and for a step that already succeeded the SDK injects the result into the return value instead of running the code.

Determinism is the price of replay, and an LLM cannot pay it

Replay only works if the code around the steps makes the same decisions in the same order every time. Temporal states the constraint directly: any time your workflow code is executed it must make the same API calls in the same sequence given the same input, and non-deterministic operations such as API calls, LLM invocations and database queries belong in Activities. People bounce off this rule because it looks arbitrary until you internalise the asymmetry: the orchestration path is executed many times, the steps are executed once.

// Wrong: the orchestration path branches on values that change on replay.
if (Date.now() - startedAt > 3_600_000) return giveUp();  // replay disagrees
const shard = Math.floor(Math.random() * 4);              // replay disagrees
const fx = await fetch(RATES_URL).then((r) => r.json());  // replay re-calls

// Right: replay-stable equivalents, or push the call into a step.
const now = await ctx.date.now();            // millis, consistent across retries
const shard = Math.floor(ctx.rand.random() * 4);   // seeded by the invocation id
const fx = await ctx.run("fx-rate", () =>
  fetch(RATES_URL).then((r) => r.json()),
);

await ctx.sleep({ seconds: 3600 });  // a durable timer, not setTimeout:
                                     // the process may not exist in an hour

// An LLM call can return a different answer to the same prompt, so it can
// never sit in the deterministic path. It is a step by construction.
const answer = await ctx.run("classify", () => llm.classify(doc));

Notice what that makes a model call. An LLM can return a different answer to the same prompt, so it can never live in the deterministic path; it is a step by construction. That is convenient rather than restrictive. The thing you most want to avoid re-running is exactly the thing the runtime forces you to isolate, which means a correctly written durable agent gets cost recovery as a side effect of satisfying the determinism rule.

Four runtimes, and where each one keeps your state

The interesting difference between these products is not the API. They converge on the same idea of wrapping the side effect. The difference is what you have to run and where the journal lives, and that is what decides whether adopting one is an afternoon or a quarter.

DBOS is the outlier worth noticing if you already run Postgres. Its architecture page is explicit that there is no separate orchestration server and no infrastructure required besides Postgres, because the library checkpoints workflows and steps into a system database inside your own cluster. Temporal is the opposite trade: a service to operate, in exchange for the deepest story on visibility, versioning and polyglot workers. Restate sits between them, with the server as the only new process and handlers that remain plain HTTP services.

Waiting three days for a human is the same primitive as waiting for a timer

This is where the ERP work and the agent work turned out to be one problem. In hierarchical-approval an instance legitimately sits pending until a person acts, slaDeadlineDays defines when the escalation scheduler should care, and the engine accepts a Clock so those days can be simulated. A test advances a ManualClock by three days, ticks the scheduler, and the SLA breach event fires with zero wall-clock time elapsed. I built that because there was no other way to test a three-day deadline in CI.

// hierarchical-approval: the engine takes a Clock, so "three days" is testable.
import { ApprovalTestKit } from "hierarchical-approval/testing";

const { engine, clock } = ApprovalTestKit.create();  // MemoryAdapter + ManualClock

await engine.defineTemplate({
  name: "purchase-order",
  documentType: "purchase_order",
  levels: [{ level: 1, name: "Finance", approvers: [finance], mode: "any" }],
  slaDeadlineDays: 2,
});

clock.advanceDays(3);               // no real timers, no three-day test run
await engine["escalation"].tick();  // 'approval:sla_breached' fires here

// A durable runtime hands you the same two primitives, first-class:
const { id, promise } = ctx.awakeable();          // one-shot token, like an SLA row
await ctx.run(() => emailApprover(instanceId, id));
const decision = await promise;   // suspends the invocation; survives a restart

// Or a named signal, which can be resolved more than once — the same channel
// works for "approved" and for steering an agent mid-run.
const steer = await ctx.signal("steer");

A durable runtime gives you those two primitives as language-level constructs instead of as something you build. A durable sleep is a journal entry the server owns, so it outlives the process that started it. An awakeable hands an external system a one-shot token to resolve, and Restate's own table lists agent steering and human approvals as precisely what signals are for. An agent pausing for sign-off and an approval pausing for a director are the same shape; I had written a narrow, domain-specific version of the general thing without knowing there was a general thing.

The costs the journal quietly adds

Three of these surprised me, and all three are documented by the vendors themselves rather than discovered in anyone's incident review. None of them is a reason to avoid durable execution, but each one is a reason to read the runtime's docs before the agent is in production rather than after.

• Long steps need the timeout raised. Restate suspends an invocation and retries it if it receives no new journal entry for more than a minute by default, and its docs call out an LLM call taking up to three minutes as the case where you must adjust the service's inactivity and abort timeouts.
• Everything a step returns is serialised into the journal. Returning a whole scraped page from a tool step writes that page into the log; returning the three extracted fields does not. I now treat a step's return type as a storage decision, because it is one.
• Deploying became a versioning problem. Temporal names non-deterministic code changes, shipping new workflow code while executions started on the old code are still running, as one of the two main sources of non-determinism, and Restate warns that a long sleep means keeping the deployment version alive until the invocation completes.

The journal is a real artefact with a size, a schema and a compatibility surface, and it now sits between your deploy pipeline and your running agents. Treat it the way you treat a database migration, not the way you treat a log file.

When you do not need any of this

The unfashionable part, stated plainly: most agent calls should never go near a workflow engine. A single tool call that returns in five seconds has nothing to resume. If it fails, run it again and you have lost five seconds and one model call. Durable execution earns its cost only when a run is long enough to be interrupted, expensive enough that repeating it hurts, or paused on something outside your own process. Here is the ladder I now climb, and I stop at the first rung that holds.

1. An idempotency key and a steps table in Postgres. Record each completed step with its output, and skip the ones already recorded. This is durable execution's core idea with none of its infrastructure, and it is enough for a three-step pipeline.
2. An in-process library. DBOS runs inside your application and needs nothing but the Postgres you already have, which makes it the smallest genuine step up from hand-rolling the steps table.
3. A hosted orchestrator. Inngest keeps the state on its side and asks only that your app expose an HTTP endpoint, so nothing new joins your estate to be patched, backed up and monitored.
4. A full workflow platform. Temporal, when you have many long-lived workflows, more than one language, and somebody whose job description includes operating it.

For a two-person shop, that fourth rung is a real bill paid in attention: a service to run, a versioning discipline to hold, and a new class of failure to learn to read. I would not pay it for one agent. I would pay it for a fleet, and the honest signal that you have a fleet is when two different teams have independently written their own steps table.

The rule I carry out of this is about granularity, not about products. Ask what your unit of retry is. If the answer is the whole job, every failure charges you again for the work that already succeeded, and with agents that work is metered by the token. Make the step the unit, with a journal, a steps table, or whatever matches the size of the problem, and a crash stops being an invoice.