Reliability & exactly-once

"Exactly-once" is an easy claim and a hard contract. This page explains exactly what rflow guarantees, how, and what it deliberately does not promise.

The claim gate: one trigger occurrence, one run

Every trigger occurrence has a deterministic key — for events, chain_id:tx_hash:log_index (plus the phase for run_on: both triggers); for cron, the scheduled tick; for read/block triggers, the poll instant / block number. Starting a run is an

INSERT ... ON CONFLICT DO NOTHING

against a UNIQUE (workflow_name, trigger_key) constraint in Postgres. Whoever inserts the row owns the run; everyone else (a replayed block range, a reorged duplicate, an overlapping backfill, a second delivery from the indexer) is a no-op. There is no window where two runs exist for one occurrence, because the database is the arbiter.

One deliberate exception: webhook triggers without idempotency_key are at-least-once — every accepted POST claims a fresh uuid key, so a caller's retry is a new run. Webhooks have no inherent dedupe identity; declare one with trigger.webhook.idempotency_key (e.g. ${{ headers['x-idempotency-key'] }} or ${{ body.event_id }}) and a duplicate POST within the TTL window creates no new run.

The journal: state before side effects

Every run and every step is journaled in Postgres (rflow.workflow_runs, rflow.step_runs). The invariant: state is persisted before the side effect. A step's row exists before its HTTP call fires or its transaction is queued, and its outcome (output, tx ids, error) is persisted when it settles.

So after a crash, recovery knows precisely where every run was:

steps that completed are never re-executed — their journaled output feeds later steps' expressions (including each foreach iteration, which journals under its own <id>[<i>] row)
a run parked in waiting_delay resumes its countdown against the persisted wake time
a run parked in waiting_tx resumes polling the same transaction
a run parked in waiting_event (wait_for) keeps its persisted wait rows and timeout deadline
a run parked in waiting_approval keeps its pending approval — the gate is a database row, not process memory

Sends: idempotency keys, not hope

The dangerous window is a send: rflow asks the relayer to queue a transaction, and the process dies before the response lands. Did the tx get queued or not?

rflow journals a client-generated idempotency key before the request:

external_id = rflow:{run_id}:{step_id}:{attempt}

The embedded rrelayer stores it under a unique index. On recovery:

look up the external_id — if the transaction exists, adopt it and keep waiting;
if it doesn't, re-send with the same external_id — and if the original was actually in flight, the unique index makes the duplicate insert fail, so the retry collapses into a safe no-op.

Either way there is exactly one live transaction for that step attempt. This is why the key must exist client-side before the request — rrelayer's own tx id only comes back in the response, which is exactly what the crash may have eaten.

After queueing, the stable tx_id is adopted for status polling, and gas bumping/rebroadcast (which change the tx hash, never the tx_id) are entirely rrelayer's.

Compensation sends inside on_reorg: carry the same discipline under their own key shape — rflow-reorg:{run_id}:{fork_block}:{step_id}:{attempt} — journaled in the response's step journal (rflow.reorg_step_runs) before the relayer can see the transaction.

Approvals: the gate is a row

An approval-gated send parks the run with a persisted approval row. rflow approve/reject (or a timeout) settles that row; the executor picks the decision up and continues. A crash while parked loses nothing — the pending approval, its expiry, and the prepared transaction summary are all journaled, and an approved send still goes through recheck + re-simulation before broadcast. What is not guaranteed: with on_timeout: proceed, an unanswered gate broadcasts — that is opt-in and rflow validate warns about it.

Sagas: wait_for guarantees

A wait_for step persists its event conditions and timeout deadline before parking, so sagas survive restarts. Every awaited (contract, event, network) gets indexed: pairs no declared trigger watches receive their own wait-only subscription at boot — it routes deliveries only to the wait matcher (it can never create a run) and starts at the latest block, catching future events while runs are parked. Resume depth is the per-condition confirmations: field — the default stays reorg-safe (a confirmed-depth delivery settles; a head delivery only arms the wait until the network's depth is reached), and confirmations: 0 opts into head-speed resumes with the reorg risk validate warns about. wait_for also parks per foreach item and inside finally: blocks, and on_timeout: goto crash-resume is exact: the jump decision is journaled with the settle, so a restart lands on the same named step.

Recovery semantics

On every boot, before triggers start flowing:

an advisory lock on (database, project) is taken — a second rflow process for the same project exits instead of double-running (or waits, if it is an HA standby)
incomplete runs (queued / running / waiting_tx / waiting_delay / waiting_event / waiting_approval) are scanned
waiting_tx steps reconcile against the relayer by external_id
elapsed delays fire, in-flight steps resume from the journal
incomplete on_reorg: responses (rflow.reorg_runs rows still running) resume through their own step journal — completed compensation steps never rerun, interrupted compensation sends reconcile by their idempotency key

Then live triggers resume from their persisted cursors.

An HA takeover runs exactly this path: the promoted standby recovers the dead leader's in-flight work first, engines and triggers after — the same exactly-once guarantees as any restart.

Replays and tests never touch live state

rflow replay and rflow test run under namespaced workflow names (replay_<session>__<wf> / test_<uuid>__<wf>) with prefixed trigger keys, so a rehearsal can never collide with a live claim, a live executor never picks up rehearsal runs, and the workflow's own run history stays clean. Dry-run mode additionally stops every send before the relayer hand-off. Sessions are reclaimable with rflow replay prune, which only ever deletes namespaced rows — never a live workflow.

Failed runs: the dead letter queue

A run that fails terminally (with on_failure: dead_letter, the default) keeps its full journal — every step's status, output, error, attempts and tx links — and lands in the DLQ. rflow runs list --failed shows them; rflow runs retry <id> re-fires one after you've fixed the cause. Repeated failures can trip a circuit breaker that pauses the workflow and pages someone.

Durable spend budgets survive restarts and races

Two workflow guards are Postgres-backed so they hold across a restart or an HA takeover:

rate_limit.durable: true persists the window in rflow.rate_limit_buckets as a fixed window claimed with one atomic INSERT … ON CONFLICT statement. A fresh process cannot reset a live window, and concurrent runs racing one bucket can never admit more than max. (The default rate_limit is in-process and resets on restart — see the caveat below.)
budgets: cap cumulative spend inside a rolling window. The cap is enforced by a reservation written before the send broadcasts, so racing runs cannot overshoot it.

A budget reservation follows the same money-path discipline as a send — persist before the side effect, key by the send's idempotency external_id, let the journal drive recovery:

Reserve — after every other pre-broadcast check, right before the relayer call, the send writes a durable rflow.budget_reservations row (one per asset) and atomically checks (live reservations + settled spend) + amount ≤ cap under a per-(budget, asset) lock. Over the cap ⇒ the send fails budget_exceeded, any partial rows for the send are released, and nothing broadcasts.
Settle or release by outcome — a send that reaches the relayer flips its reservations to spent. A failure that proves the send never left the wallet (insufficient funds, a 429 rate-limit) flips them to released (refunded) immediately. An ambiguous failure (an rpc timeout, an unknown transport error) is instead left reserved: the transaction may have landed, so releasing it inline would drop it from the cap and let a concurrent send slip past — the reconcile settles it exactly-once. A send that is queued but then fails on-chain (reverted / dropped / expired) moved no value, so its already-spent reservation is refunded to released too — so a failed send never permanently consumes budget and its retry is not falsely refused.
Reconcile — a crash between reserve and settle, or an unresolved ambiguous failure, leaves live reserved rows. The recovery pass reconciles them by external_id exactly like a send: one that provably landed ⇒ spent (reclaiming even a prematurely released row), one that never landed ⇒ released. The money is accounted at most once, and a queued value is treated as committed spend (conservative for a cap).

Because the reservation is keyed by external_id, a re-run of the same attempt reserves the same row — a double reserve is one row, not double spend. See Budgets for the config surface and rflow budgets ls.

What shutdown does — and does not — guarantee

On Ctrl-C / SIGTERM, rflow stops accepting new work, lets in-flight steps reach a journal-consistent point, and shuts the engines down (the relayer drains its queues). Combined with recovery, a kill -9 at any point is safe in the sense that matters: no duplicated side effect and no lost run.

What is not guaranteed:

At-most-once for non-idempotent HTTP. A crash after an http_call fired but before its outcome was journaled means recovery re-executes that step. Sends are protected by idempotency keys; arbitrary HTTP cannot be. Use the hmac signature plus an idempotency key of your own (e.g. run.id) on the receiving side if your endpoint is not idempotent. The same applies to a notify: in that crash window — you might get one alert twice. You will never get zero.
On-chain finality. rflow's exactly-once is about its own actions. A reorg can still orphan the block your transaction landed in — see Reorgs — and can orphan the triggering event after a head-fired run already acted. confirmations: is the knob for that, and on_reorg: is the response hook (itself exactly-once per (run, fork) and crash-resumable through its own step journal — see the reorgs page).
Missed cron ticks. By default, a cron tick that comes due while the process is down is skipped. Opt into catch_up: true to replay missed slots on boot — capped at the newest 100. Transient claim errors while the process is up are retried within the tick's window (the dedupe key is the scheduled time, so retrying can never double-fire).
Per-instance windows. rate_limit windows are in-process by default — a restart or takeover starts them fresh. Set rate_limit.durable: true for a Postgres-backed fixed window that a restart cannot reset. (Trigger throttle:, spend budgets and circuit-breaker state are Postgres-backed and survive restarts.)

Where state lives

Everything is in one Postgres (schemas rflow, relayer, rindexer). Back that up and you can rebuild a machine from scratch: the YAML is the definition, Postgres is the memory. Nothing else — no local files rflow can't regenerate (.rflow/ is generated runtime config, safe to delete).