Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

State inspection & repair — supported recovery, not manual SQL

When durable state gets stuck (a saga parked forever on a wait_for:, a tripped circuit breaker, an unanswered approval, a cursor that needs re-scanning), rflow has supported, journal-aware repair commands. No manual SQL.

Two tiers:

  • Read-only inspection: rflow state inspect plus cursors ls / waits ls / delays ls / circuit ls / approvals ls.
  • Narrow repair mutations: one command per known stuck state, each with a plan printed first, a required --yes, and an audit row written in the same transaction as the fix.

Three guarantees:

  1. A repair can never create a duplicate send or corrupt exactly-once state. No repair touches a run with an in-flight send (waiting_tx). Terminalizing a stuck wait fails the run; it never re-runs a money-moving step with a new idempotency key.
  2. Terminalize, never delete. Repairs settle stuck work to a terminal status; they never delete run history or edit a step's output.
  3. Every mutation is auditable. Each applied repair writes one rflow.operator_audit row (action, target, args, reason, actor, created_at) in the same transaction as the change; rflow archive export can dump it.

Each mutation is one transaction that re-checks its precondition under a row lock (no TOCTOU): a precondition that no longer holds is refused, changing nothing, which also makes every command idempotent under a re-run.

rflow state inspect

The durable-state dashboard. Read-only; every query is a plain SELECT.

rflow state inspect                      # everything
rflow state inspect --workflow treasury-sweep
rflow state inspect --json               # machine-readable snapshot

It reports run counts by status, parked saga waits, parked delays (with wake_at), pending approvals, trigger cursors, circuit-breaker state, replay/test session leftovers, and a tail of recent operator repairs. The scoped read-only listings back the same data:

rflow cursors ls [--workflow <wf>]
rflow waits ls   [--workflow <wf>]
rflow delays ls  [--workflow <wf>]
rflow circuit ls [--workflow <wf>]
rflow approvals ls [--all]

Repair commands

Every repair prints its plan and refuses without --yes:

$ rflow circuit reset treasury-sweep
PLAN: reset circuit breaker 'treasury-sweep'
  current state: open
action: close the breaker, clear its failure window, and resume the workflow if the breaker paused it (an operator's own pause stands).
refusing to reset without --yes. This is the plan; re-run with `--yes` to apply it.

rflow cursors set <workflow> <network> --block <n> --yes

Moves a trigger cursor's last-processed block to re-scan or skip ahead.

  • Lowering re-scans the range. Events already claimed dedupe by the workflow_runs (workflow_name, trigger_key) UNIQUE constraint, so a re-scan cannot create a duplicate run or a duplicate send: only genuinely unclaimed matches in the range fire. The plan warns loudly.
  • Raising skips the range: unprocessed events in it are permanently skipped.
  • Refused below the deployment anchor, or when already at the target block.

rflow waits cancel <run-id> --reason "<why>" --yes

Cancels a saga stuck on a wait_for: by terminalizing the run to failed with your reason and dropping its open event-wait rows. History is never deleted. Refused if the run is not actually parked on a wait (waiting_event), which includes any run holding an in-flight send.

rflow delays wake <run-id> --yes

Wakes a parked durable delay early by setting the delay step's wake_at to now. A running engine keeps the run parked in memory until its original wake time and never re-scans waiting_delay runs, so the early wake is applied on the next engine restart / failover recovery pass; restart or fail over to resume it now. It only re-arms the durable timer; the resume path is exactly-once by external_id, so no send is duplicated. Refused if the run is not waiting_delay, or if the delay is already due.

rflow approvals expire <id> --yes

Expires a pending approval so the run takes its approval on_timeout path, exactly as a natural timeout would. The command itself never broadcasts, but it does not cancel the send: the running engine then applies on_timeout, and that depends on the mode.

  • on_timeout: fail (the default): the parked send is dropped and never broadcast.
  • on_timeout: proceed: the timeout is treated as consent, so the running engine BROADCASTS the parked send. The plan warns loudly in this case; expiring is not a way to abort a proceed gate.

Refused if the approval is not pending.

rflow circuit reset <workflow> --yes

Closes a tripped breaker, clears its failure window, and resumes the workflow, but only if the breaker itself paused it (paused_by = circuit_breaker). An operator's own rflow workflow pause during the incident always stands. Refused if the breaker is already closed.

Doctor points you at the fix

rflow doctor --deep detects these stuck states and names the exact repair command in its hint: a stale cursor → rflow cursors set, a stuck wait past its timeout → rflow waits cancel, a tripped breaker → rflow circuit reset, a stale pending approval → rflow approvals expire.

Non-goals

There is no general SQL console, no arbitrary step-output editing, and no replaying of completed money-moving steps with new idempotency keys. Those are exactly the operations that would break exactly-once, so they are out of scope by design.