Durable Execution: How Temporal-Style Replay Works

There is a table in your order service called workflow_state, and it has been lying to you for two years. It started honest: an enum with four statuses and an updated_at. Then payments got a retry counter. Then someone added next_retry_at and a sweeper cron that re-drives rows stuck in CHARGING. Then a second cron, because the first one double-charged a customer when it raced a slow webhook, so now there's a sweeper_lock column too. The enum has eleven states. Two of them are misspelled and load-bearing. The runbook page titled "orders stuck in RESERVING" has nine revisions and a hand-drawn state diagram that disagrees with the code.

None of this is incompetence. It's what happens when you implement a correct distributed workflow by hand: the saga orchestration, the outbox table with CDC so events publish atomically with commits, the retry scheduler, the idempotency bookkeeping. Five patterns, each individually reasonable, composed into a machine that only three people understand.

Durable execution is the observation that this machine is generic and someone else should ship it. It is not magic, and the vendors selling it do it no favors by describing it as "crash-proof code." The mechanism is precise and worth knowing exactly, because its two big constraints — determinism and versioning — follow directly from it, and both will page you if you learn them the hard way.

Event sourcing for your call stack

A durable execution engine splits your program into two kinds of code.

Workflow code is orchestration: the control flow that decides what happens in which order. It looks like an ordinary function, and that's the sales pitch — but it runs under unusual rules we'll get to.

Activities are everything that touches the outside world: HTTP calls, database writes, sending email, invoking an LLM. An activity is ordinary code with no special rules. The engine wraps it with retries, timeouts, and heartbeats.

Here's a provisioning workflow in Temporal-style TypeScript:

// Workflow code: orchestration only, deterministic by contract.
export async function provisionCustomer(order: Order): Promise<void> {
  const account = await acts.createAccount(order);        // activity
  const charge  = await acts.chargeCard(order.paymentId); // activity
  await acts.grantEntitlements(account.id, order.sku);    // activity
  await sleep('30 days');                                 // durable timer
  await acts.sendRenewalReminder(account.id);             // activity
}

When this runs, the worker doesn't just execute it. Each await on an activity emits a command to the engine — "schedule chargeCard" — and the engine appends events to a per-execution event history:

 1  WorkflowExecutionStarted   { input: order #8412 }
 2  ActivityTaskScheduled      { createAccount }
 5  ActivityTaskCompleted      { result: { accountId: "acc_91" } }
 6  ActivityTaskScheduled      { chargeCard }
 9  ActivityTaskCompleted      { result: { chargeId: "ch_812", amount: 4900 } }
10  ActivityTaskScheduled      { grantEntitlements }
13  ActivityTaskCompleted      { result: ok }
14  TimerStarted               { fireAfter: 30d }
        ... 30 days pass, no process is waiting ...
15  TimerFired
16  ActivityTaskScheduled      { sendRenewalReminder }

That history — persisted by the engine, replicated, durable — is the source of truth for the execution. Per Temporal's docs, it is "an ordered log of everything that has already happened in a Workflow," and state is restored by replaying it, not by snapshotting process memory. The worker holding your function's stack frame is cattle. Kill it whenever.

sequenceDiagram
    participant C as Client
    participant S as Engine
    participant W as Worker
    participant P as Payment API
    C->>S: start provisionCustomer(order)
    S->>S: append WorkflowExecutionStarted
    S->>W: workflow task
    W->>S: command — schedule chargeCard
    S->>S: append ActivityTaskScheduled
    S->>W: activity task
    W->>P: POST /charge
    P-->>W: ch_812
    W->>S: activity result
    S->>S: append ActivityTaskCompleted
    S->>W: next workflow task with updated history

If you've read the event sourcing article, this should feel familiar: current state = fold(initial, events). The difference is what's being folded. Event sourcing rebuilds a domain aggregate — an account balance — from domain events. Durable execution rebuilds a call stack — your function's local variables and program counter — from execution events. Same log, same fold, different subject.

Notice what the sleep('30 days') costs while waiting: one TimerStarted event and a row in the engine's timer store. No parked thread, no pod pinned for a month, no cron. This is why durable execution owns the "subscription renewal in 30 days" and "escalate the ticket if no human replies in 48 hours" problem class.

Replay: the code re-runs, the world does not

Now the interesting part. The worker running provisionCustomer gets OOM-killed between events 13 and 14 — entitlements granted, timer not yet started.

The engine notices (workflow task timeout), and puts a workflow task on the queue. A different worker picks it up, loads the history, and does something that sounds insane the first time you hear it: it runs the function again, from line one.

But the SDK is intercepting every await. createAccount — is there a journaled result for the first activity? Yes, event 5. Return { accountId: "acc_91" } immediately, without scheduling anything. chargeCard — event 9, return ch_812. No second charge. grantEntitlements — event 13, return ok. Now the code reaches sleep('30 days') and the history has no more events to feed. Replay is over; the worker emits the "start timer" command for real, and the execution is live again — with account and charge sitting in local variables, rebuilt by re-execution rather than restored from a snapshot.

sequenceDiagram
    participant S as Engine
    participant W2 as New worker
    Note over S: original worker died after event 13
    S->>W2: workflow task + full history
    W2->>W2: run provisionCustomer from the top
    W2->>W2: await createAccount — replay feeds event 5
    W2->>W2: await chargeCard — replay feeds event 9, no re-charge
    W2->>W2: await grantEntitlements — replay feeds event 13
    W2->>S: history exhausted — command: start 30-day timer
    S->>S: append TimerStarted
    Note over S,W2: execution is live again, nothing ran twice

The code re-runs; the world does not. That one sentence is the entire mechanism, and every property of these systems falls out of it. Restate's definition is the same mechanism in different clothes: "each previously-completed step returns its recorded result instantly, until execution catches up to the point of failure and continues from there."

In practice engines avoid replaying on every step. Temporal keeps a sticky queue per worker: as long as the same worker stays alive, it keeps the workflow's state cached in memory and only receives new events. Full replay happens on crash, deploy, or cache eviction — which is exactly when you need it.

Two consequences worth internalizing before we hit the constraints:

Workflow code gets exactly-once effect, not exactly-once execution. The function body may run dozens of times across replays. What happens exactly once is the journaled decision sequence — which is all that matters, because effects live in activities.

Activities are at-least-once. If a worker finishes chargeCard but dies before reporting the result, the engine sees a timeout and schedules a retry. The card gets charged twice unless chargeCard carries an idempotency key. The engine dedupes its own bookkeeping, not your payment processor's ledger. Everything in the idempotency article still applies inside activities — durable execution moves the retry loop into the platform, not the responsibility.

Why workflow code must be deterministic

Replay only works if the re-executed function makes the same decisions in the same order — if it emits the same command sequence the original run journaled. The moment it doesn't, the SDK is holding a command ("schedule fraudCheck") that doesn't match the next history event ("ActivityTaskScheduled: reserveInventory"), and it has no idea how to reconcile your code with reality. Temporal calls this a non-determinism error, and its docs are blunt about the cause: workflow code "can not have inline logic that branches (emits a different Command sequence) based off a local time setting or a random number."

So inside workflow code, the following are banned:

• Wall-clock time. Date.now() differs between original run and replay. Use the SDK's workflow time, which returns journaled time.
• Randomness and UUID generation. Same problem. SDKs provide seeded, journaled equivalents.
• Naked I/O. Any network call, file read, or DB query would re-execute during replay — the exact thing the architecture exists to prevent. Effects go in activities.
• Iteration over unordered collections where order affects which commands you emit. A Go map range or Python set iteration can reorder between runs.
• Threads, environment variables, global mutable state. Anything that can differ across processes.

The SDKs work hard to make the safe path the easy path. Temporal's TypeScript SDK runs workflow code inside a V8 isolate with Date.now and Math.random swapped for deterministic versions; the Python SDK runs workflows in a sandbox that flags non-deterministic imports. But no SDK can catch a branch on if (order.total > threshold) where threshold comes from a config file read at module load — that's a landmine that detonates on the next deploy, not in code review. The discipline is simple to state: workflow code computes; activities act. Anything you couldn't compute twice identically belongs in an activity, where the result gets journaled and replay feeds it back.

Note what this makes suddenly fine: nondeterministic services. An LLM call is wildly nondeterministic — and completely safe inside an activity, because replay never re-invokes it. This detail is why durable execution and AI agents fit together, and we'll come back to it.

The versioning trap

Here is where teams get hurt in production. Determinism must hold not just across replays of the same binary, but across deploys.

Say v1 of the workflow is charge → reserve → email, and there are 40,000 in-flight executions journaled against it. You ship v2, which inserts a fraud check: charge → fraudCheck → reserve → email. A v1 execution that crashed after charge now replays on v2 code. The code's second command is "schedule fraudCheck." The history's next event says reserve was scheduled. Mismatch. Non-determinism error. The workflow task fails and retries forever, and your dashboard fills with stuck workflows — not new traffic, but every old execution that happens to need a replay.

flowchart TD
    H["In-flight history from v1\ncharge, reserve journaled"] --> R{"Replay on v2 code\ncharge, fraudCheck, reserve"}
    R -->|"commands match history"| OK["Resume normally"]
    R -->|"v2 emits fraudCheck where\nhistory says reserve"| ND["Non-determinism error\nworkflow task stuck, retrying"]
    ND --> FIX["patched() branch, pinned\nworker versions, or reset"]
    style OK fill:#22c55e,color:#111
    style ND fill:#dc2626,color:#fff
    style FIX fill:#0e7490,color:#fff

The engines give you two escape hatches, both with real costs:

Patching. You branch in code on a journaled marker: if (patched('add-fraud-check')) { await acts.fraudCheck(...) }. New executions journal the marker and take the new path; old histories replay without it and take the old path. It works, and it turns your workflow into an archaeology site — every live change accumulates a branch you can only delete (via a deprecation step) once no execution from the old era can ever replay again. For workflows that run for months, "the old era" lasts months.

Worker versioning. Pin executions to the code revision that started them: old workers drain old runs, new workers take new runs. Temporal recommends this direction — it keeps the code clean at the price of running N worker fleets while long-lived executions drain.

The staff-level framing: workflow code is a wire format. You wouldn't change a protobuf field's meaning and redeploy against live traffic; don't change a workflow's command sequence against live histories. Put a replay test in CI — Temporal ships a replayer that runs recorded production histories against your new code and fails the build on divergence. Teams that treat workflow diffs like schema migrations do fine. Teams that discover this doc page during an incident do not.

The stack you no longer hand-roll

Go back to the workflow_state table from the opening. Map what each hand-rolled piece was doing onto what the engine now owns:

Hand-rolled piece	In a durable execution engine
status column + state enum	The event history is the state machine
Outbox table + CDC publisher	Gone — activity scheduling is journaled atomically by the engine
Retry columns + sweeper cron	Per-activity retry policies with backoff, built in
Scheduled-job service for timeouts	Durable timers — sleep('30 days') is one event
Resumption logic after crashes	Replay
"Where is order #8412?" forensics	Query the history — every step, input, and result is journaled
Compensation logic	Still yours — a try/catch in workflow code issuing compensating activities
Activity idempotency	Still yours

The two "still yours" rows matter. Durable execution does not replace the saga pattern — sagas are about what to do when step 3 of 5 fails (compensate semantically, because steps 1–2 already committed). The engine replaces the plumbing a saga needs: it guarantees your compensation code actually runs to completion even if the orchestrator dies mid-compensation, which is precisely the part of a hand-rolled orchestrator that's hardest to get right. In a payment system, that's the difference between "the refund logic exists" and "the refund logic survives the node dying halfway through issuing it."

So here's the position this article is committed to: if you're about to build a multi-step business flow with real side effects, evaluate a durable execution engine before hand-rolling saga + outbox + retry-scheduler. The hand-rolled stack is three or four distributed-systems patterns composed by hand, each a place to introduce the double-charge bug. The engine is one dependency that ships all of them tested. The honest counterweight comes two sections down — the overhead is real, the operational burden is real, and there are paths it doesn't belong on.

The engines, honestly

Mechanism first: everything above — journaled history, deterministic replay, activities, the versioning trap — is stable across vendors and will still be true in five years. What follows is product-specific and volatile; treat all of it as "as of mid-2026."

Engine	Where the journal lives	Model	Honest one-liner
Temporal	Separate server cluster (Cassandra/Postgres backing) or Temporal Cloud	Workflows + activities in Go/Java/Python/TS/.NET; full replay	The default choice and the most battle-tested; heaviest to self-host
Restate	Single Rust binary owning a replicated log + state store, fronting your services	Durable handlers + virtual objects over HTTP	Lightest architecture per unit of capability; young ecosystem
DBOS	Your existing Postgres — workflows and step outputs as rows	Decorated Python/TS/Go/Java functions, checkpoint per step	Smallest possible on-ramp; ceiling is your Postgres
Inngest	Managed (or self-hosted) state store; functions re-invoked over HTTP per step	step.run() memoization, event-triggered	Serverless-native and pleasant; you're inside its event model
AWS Step Functions	AWS-managed state machine execution history	JSON state machine (ASL), not code	Great glue for AWS services; miserable for complex logic, cost scales per transition

Temporal is the reference implementation of the model in this article — its lineage runs through Uber's Cadence and AWS Simple Workflow, built by the same people. It is also, as of early 2026, the commercial proof that this category is real: a $300M Series D at a $5B valuation with OpenAI named as a production customer and the platform absorbing spikes above 150,000 actions per second. Self-hosting the cluster is a genuine operational commitment; that's what the Cloud product is for, and Replay 2026 added serverless workers on AWS Lambda (pre-release) to shrink the on-ramp further.

Restate rebuilt the stack from first principles as a single binary that acts as a durable proxy in front of your services, journaling invocations to its own replicated log. Its pitch is latency — a published p99 under 100 ms for a 10-step workflow — and operational simplicity. The mechanism is the same journal-and-replay; the packaging is what differs.

DBOS takes the position that you already run a durable, transactional, replicated store: Postgres. Workflows are rows; each step's output is checkpointed to a table, and a step that is a database transaction can commit its business write and its checkpoint in the same transaction — genuine exactly-once for DB steps, no journal round trip to a separate system. The trade is that your workflow throughput and your application share a database.

Inngest inverts the worker model: instead of long-lived workers polling a server, the platform re-invokes your function over HTTP for each step, memoizing completed step.run() results and injecting them on re-entry. Same fold, delivered as serverless-friendly middleware. The determinism constraint survives in softer clothing: anything nondeterministic must live inside a step.run().

Step Functions is what you get when the journal idea is implemented as a hosted JSON state machine rather than code. It's genuinely good at wiring Lambdas and AWS services with retries and it never pages you. But expressing real control flow in Amazon States Language JSON gets painful fast, the Standard-workflow history caps at 25,000 events, and the pricing meter runs per state transition — do the math in the next section before committing an order pipeline to it.

Also in the family: Cadence (Temporal's still-maintained ancestor at Uber) and Azure Durable Functions (the same replay trick bolted onto Azure Functions).

What durable execution costs

The engine buys correctness with journal writes. Two budgets to check before adopting: latency and history volume.

Per-step overhead (illustrative, mid-2026):
  Temporal-style engine, same region:  ~5–20 ms per activity
    (schedule → journal → dispatch → complete → journal)
  DBOS, checkpoint in same-AZ Postgres: ~1–5 ms per step
  Restate, vendor-published:            p99 < 100 ms for a 10-step workflow

A 10-step order flow:
  bare service calls:            10 × ~2 ms            ≈  20 ms
  durable @ ~10 ms/step:         10 × (2 + 10) ms      ≈ 120 ms
  → invisible in checkout, provisioning, refunds (human-facing, seconds-scale)
  → disqualifying on a read path with a 25 ms p99 budget

History volume at scale:
  1M workflow executions/day × ~20 events × ~2 KB/event ≈ 40 GB/day
  → retention policy and archival tier are day-one decisions, not cleanup

Step Functions sanity check (us-east-1 list price, as of mid-2026):
  1M executions/day × 12 transitions = 12M transitions/day
  12M × $0.000025 ≈ $300/day ≈ $9,000/month — before Lambda charges

The latency line is the judgment call the vendors won't make for you: durable execution is priced per decision, so it belongs on flows where each decision is worth milliseconds — a payment moving, an agent acting. It does not belong between your API gateway and your database on the hot read path, and no amount of engine tuning changes that arithmetic, because the journal write is the product.

What breaks

The failure modes cluster around the mechanism, which is exactly why you learned the mechanism.

Non-determinism on deploy. Covered above, listed again because it is the most common production incident with these systems: a workflow diff ships without a patch marker, and every in-flight execution that replays against it stalls. The tell is a spike in workflow-task failures right after a deploy with no increase in traffic. Prevention is boring and works — replay tests in CI, patches for every behavioral change, worker pinning for long-lived flows.

History growth. Temporal terminates an execution at 51,200 events or 50 MB, warning at 10,240 events or 10 MB. That sounds enormous until you write a polling loop: a timer every 10 seconds is ~8,600 iterations/day, each costing a handful of events — you're dead inside two days. Long-lived and looping workflows must call continue-as-new, which closes the run and atomically starts a fresh one (same workflow ID, empty history), carrying state forward as arguments. Entity workflows — one execution per customer, alive for years — live and die by scheduling their continue-as-new correctly.

Fat payloads. Every activity input and result goes into the history. Pass a 5 MB rendered PDF as an activity result and ten steps later you've hit the 50 MB cap; you'll also make every replay slow. Pass references — an S3 key, a row ID — and keep payloads in blob storage. (Temporal productized this as external payload storage in 2026; the discipline predates the feature.)

"Exactly-once" wishful thinking. The engine's guarantee is exactly-once workflow effect, at-least-once activity execution. The activity that completed but whose worker died before reporting will run again. If chargeCard doesn't send an idempotency key to the payment processor, durable execution will faithfully, durably double-charge your customer — and the history will show you exactly how. The engine moved the retry loop; it didn't repeal the two-generals problem.

The engine becomes tier-0. Once real flows run on it, the engine's availability bounds everyone's. The failure mode is softer than a crashed service — executions pause and resume with nothing lost, which is the whole point — but "every workflow in the company is paused" is still a Sev-1, and self-hosting means the history store (Cassandra or Postgres under Temporal) is now among the most critical databases you run. Budget the on-call accordingly or pay for the managed tier.

One hot workflow. A single execution is logically single-threaded — its history is one ordered log. Funnel a high-rate event stream into one execution (say, all ticks for a symbol into one "market" workflow) and you'll saturate the per-execution task pipeline while the rest of the cluster idles; Temporal also caps pending signals and pending activities at 2,000 per execution by default. Shard by entity — one workflow per order, per customer, per agent session — and let the engine spread them.

Durable agents: why this is suddenly everyone's problem

The reason durable execution graduated from "workflow-engine niche" to a $5B category is that the industry started shipping software with exactly the shape these engines were built for: the AI agent loop. An agent is a long-running process that alternates reasoning (an LLM call) with tool use (side effects), sometimes waits hours for a human approval, and must not lose its accumulated state at step 37 of 50. That is a workflow, drawn with different marketing.

The mapping is nearly mechanical. The agent loop is workflow code. Each LLM invocation and each tool call is an activity — and recall the detail from the determinism section: the LLM's nondeterminism is harmless here, because the response is journaled and replay feeds it back rather than re-sampling. A crashed agent resumes mid-plan with its entire trajectory intact. Human-in-the-loop approval is a signal plus a durable timer, not a polling loop. Token streaming is the awkward fit — history events are the wrong granularity for per-token updates, which is why Temporal built workflow streams on its signal/update primitives (public preview, mid-2026) and why chatty agent UIs still route tokens around the engine, journaling only the completed step.

This is no longer speculative architecture. OpenAI is a named production customer of Temporal, whose VP of App Infrastructure framed durability as "a core requirement for modern AI systems" in the Series D announcement; integrations with the OpenAI Agents SDK and Google's ADK shipped through 2025–2026. Product specifics volatile as always — but the mechanism fit is stable, and if you're sketching an agent platform, the orchestration layer is the box a durable execution engine fills.

The decision in practice

The reach-for-it rule: three or more steps, real side effects, and a lifetime longer than one request — use a durable execution engine. Order fulfillment, payment capture and refund, user provisioning, subscription lifecycle, document pipelines, agent sessions. If you're sketching an outbox table and a sweeper cron on a whiteboard, that's the tell — you're about to hand-roll one of these, worse.

Keep it off: hot request paths with tight latency budgets (the per-step journal write is the product; it can't be optimized away), single-step operations (a queue with retries is enough), and high-volume stream transformation — enriching a Kafka stream is Flink's job, not a workflow engine's, and the sane architectures use both: events on the log as the backbone, an engine execution per entity-level process the events trigger.

Picking within the category, as of mid-2026: a small team already on Postgres should start with DBOS — it's a library import, not an infrastructure project. Teams that want the deepest ecosystem and the longest production track record pick Temporal, and unless running Cassandra sounds like fun, that means Temporal Cloud. Restate is the one to evaluate if per-step latency or single-binary ops are your constraint. Inngest fits serverless-first TypeScript shops that want durability without a worker fleet. Step Functions is right when the steps are AWS services and the logic is simple — after you've done the transition-cost multiplication and checked your longest execution against the 25,000-event cap.

And whichever you pick, the mechanism travels with you: the journal is an event-sourced log, replay is a deterministic fold, activities are at-least-once, and workflow code is a wire format. Learn it once; the vendors are just packaging.

Further reading

• Temporal: Workflows and Workflow definitions — the canonical statement of history, replay, and determinism constraints
• Temporal: Workflow execution limits — the 51,200-event / 50 MB numbers and their friends
• Restate: What is durable execution? — the same mechanism from the single-binary camp, with latency numbers
• DBOS Transact (Python) — durable execution as a library over Postgres
• Inngest: How functions are executed — step memoization as an alternative packaging of replay
• Kai Waehner: The rise of the durable execution engine — where these engines sit next to Kafka and stream processing
• On this site: Event Sourcing & CQRS, The Saga Pattern, and Idempotency & Exactly-Once — the three ideas durable execution composes