Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

State inspection & repair — supported recovery, not manual SQL

A durable workflow engine will eventually get something stuck: a bridge message that never arrives leaves a saga parked on a wait_for: forever; a bad RPC or config trips a circuit breaker; an approval nobody answers sits pending; an indexer falls behind and a cursor needs re-scanning. When that happens you need supported, journal-aware repair commands — not a psql prompt and a prayer.

rflow gives you two tiers:

  • Read-only inspection — rflow state inspect plus cursors ls / waits ls / delays ls / circuit ls / approvals ls. Look before you touch anything.
  • Narrow repair mutations — one command per known stuck state, each with explicit domain semantics, a plan printed first, a required --yes, and an audit row written in the same transaction as the fix.

Three guarantees shape all of it:

  1. A repair can never create a duplicate send or corrupt exactly-once state. No repair touches a run with an in-flight send (waiting_tx). Terminalizing a stuck wait fails the run — it never re-runs a money-moving step with a new idempotency key.
  2. Prefer terminalizing over deleting. Repairs settle stuck work to a terminal status; they never delete run history or edit a step's output.
  3. Every mutation is auditable. Each applied repair writes one rflow.operator_audit row (action, target, args, reason, actor, created_at) inside the same transaction that made the change — so the fix exists in history iff it actually happened, and rflow archive export can dump it.

Each mutation runs as one transaction that re-checks its precondition under a row lock, so the plan you saw and the change you applied can never disagree (no TOCTOU). A precondition that no longer holds is refused, changing nothing — which also makes every command idempotent under a re-run.

rflow state inspect

The durable-state dashboard. Read-only; every query is a plain SELECT.

rflow state inspect                      # everything
rflow state inspect --workflow treasury-sweep
rflow state inspect --json               # machine-readable snapshot

It reports run counts by status, parked saga waits, parked delays (with wake_at), pending approvals, trigger cursors, circuit-breaker state, replay/test session leftovers, and a tail of recent operator repairs. The scoped read-only listings back the same data:

rflow cursors ls [--workflow <wf>]
rflow waits ls   [--workflow <wf>]
rflow delays ls  [--workflow <wf>]
rflow circuit ls [--workflow <wf>]
rflow approvals ls [--all]

Repair commands

Every repair prints its plan and refuses without --yes:

$ rflow circuit reset treasury-sweep
PLAN: reset circuit breaker 'treasury-sweep'
  current state: open
action: close the breaker, clear its failure window, and resume the workflow if the breaker paused it (an operator's own pause stands).
refusing to reset without --yes. This is the plan; re-run with `--yes` to apply it.

rflow cursors set <workflow> <network> --block <n> --yes

Moves a trigger cursor's last-processed block to re-scan or skip ahead.

  • Lowering re-scans the range. Events already claimed dedupe by the workflow_runs (workflow_name, trigger_key) UNIQUE constraint, so a re-scan cannot create a duplicate run or a duplicate send — only genuinely unclaimed matches in the range fire. The plan warns loudly.
  • Raising skips the range — unprocessed events in it are permanently skipped.
  • Refused below the deployment anchor, or when already at the target block.

rflow waits cancel <run-id> --reason "<why>" --yes

Cancels a saga stuck on a wait_for: — terminalizes the run to failed with your reason and drops its open event-wait rows. History is never deleted. Refused if the run is not actually parked on a wait (waiting_event), which includes any run holding an in-flight send.

rflow delays wake <run-id> --yes

Wakes a parked durable delay early by setting the delay step's wake_at to now. A running engine keeps the run parked in memory until its original wake time and never re-scans waiting_delay runs, so the early wake is applied on the next engine restart / failover recovery pass — restart or fail over to resume it now. It only re-arms the durable timer — the resume path is exactly-once by external_id, so no send is duplicated. Refused if the run is not waiting_delay, or if the delay is already due.

rflow approvals expire <id> --yes

Expires a pending approval so the run takes its approval on_timeout path — exactly as a natural timeout would. The command itself never broadcasts, but it does not cancel the send: the running engine then applies on_timeout, and that depends on the mode.

  • on_timeout: fail (the default): the parked send is dropped and never broadcast.
  • on_timeout: proceed: the timeout is treated as consent, so the running engine BROADCASTS the parked send. The plan warns loudly in this case — expiring is not a way to abort a proceed gate.

Refused if the approval is not pending.

rflow circuit reset <workflow> --yes

Closes a tripped breaker, clears its failure window, and resumes the workflow — but only if the breaker itself paused it (paused_by = circuit_breaker). An operator's own rflow workflow pause during the incident always stands. Refused if the breaker is already closed.

Doctor points you at the fix

rflow doctor --deep detects these stuck states and names the exact repair command in its hint: a stale cursor → rflow cursors set, a stuck wait past its timeout → rflow waits cancel, a tripped breaker → rflow circuit reset, a stale pending approval → rflow approvals expire.

Non-goals

There is no general SQL console, no arbitrary step-output editing, and no replaying of completed money-moving steps with new idempotency keys. Those are exactly the operations that would break exactly-once, so they are out of scope by design.