Backtesting — replay, test & dry-run

rflow replay re-indexes historical events or generates block triggers over a range, rehearses a workflow, and reports its decisions and prepared transactions. rflow test rehearses one fixture or one real transaction for any trigger kind. Both use the workflow engine's dry-run mode by default.

At a glance:

rflow replay — a range of history

rflow replay <workflow> --from-block <n> [--to-block <n>] [--timeout <secs>] [--live] [--yes]

Replay drives event triggers (a bounded re-index of the range) and block triggers (one dry-run per block where block % every == 0, deduped by block number). Cron/webhook/read/query workflows have no block-bounded history; use rflow test with a fixture.

The outputs below are a real session from the repo's token-transfer-relay example: three historical deposits (250, 25 and 500 RFT; the workflow's where: matches deposits ≥ 100):

rflow replay echo-deposit --from-block 40 --to-block 46

dry-run replay of 'echo-deposit' from block 40 to 46 - built-in sends are suppressed; commands execute unless dry_run: skip
 
replay 20260724034558fe323d - 'echo-deposit' blocks 40..=46 on local_anvil (dry-run)
 block | tx                 | run       | steps | would send
 43    | 0xfe24bb08da25fb6… | succeeded | 2 ok  | transfer(address,uint256) -> 0x5FbDB231567…, gas 51710
 45    | 0x75a03bb563b6a7c… | succeeded | 2 ok  | transfer(address,uint256) -> 0x5FbDB231567…, gas 51710
 
2 event(s) matched, 2 run(s), 2 would-be send(s)
session journal: rflow runs list (workflow 'replay_20260724034558fe323d__echo-deposit') - live history untouched

The 25 RFT deposit at block 44 is absent: the where: filtered it, exactly as live would. Each row is one claimed run: the triggering event, per-step outcomes (2 ok / failed at <step>), and the decoded would-be send with its simulated gas.

What dry-run means, precisely

Dry-run still evaluates templates, permissions, relayer policy caps, simulation when enabled or required, assert_sim, gas caps and recheck. RPC reads and local commands still run; dry-run suppresses the native effects listed below and records the session journal in Postgres:

Dry-run also skips spend-budget reservations and the recorded gas-spend gate; it does not prove that a live send would be admitted under the budget at execution time. Approval, delay and wait shortcuts do not test real human decisions, elapsed time or live event matching.

Historical-state limits

Historical trigger payloads do not pin a step's eth_call, gas estimate or simulation to that trigger's block. read: and simulate: steps, send simulations, current gas prices and now() therefore reflect execution time. An archive-capable RPC does not change this: the engine does not supply a historical block argument for those calls. The archive preflight probe is a capability check, not evidence that a rehearsal reconstructed past state.

For a rehearsal at a specific chain state, start an Anvil fork yourself at the desired block and point the project's RPC at that fork. Isolate any SQL/state inputs as well. A fixed fork gives one snapshot; rflow does not automatically advance or rewind it for each historical trigger, nor apply simulated sends to build a hypothetical portfolio. Off-chain API responses are not reconstructed. Results from different dates or RPC/database states can differ for the same historical event range.

The isolation model

A replay boots an isolated session: a bounded indexer manifest for just this workflow and range, registered under a namespaced name (replay_<session>__<workflow>) with a replay:<session>: trigger-key prefix. Consequences:

• live dedupe is untouched: a live claim and a replay claim of the same event coexist
• a live rflow start never picks up replay runs, and vice versa
• the workflow's live run history stays clean: replay runs journal under the namespaced name
• trigger.throttle is dropped for the session (replay shows every match), the trigger is forced to confirmed-at-depth-0 (historical blocks are final), and decoded fires are never offered to live parked wait_for sagas
• paused: is ignored: replaying a paused workflow is an explicit operator action

This isolation covers workflow names, claims and native dry-run mutations. It is not a sandbox for commands, a snapshot of shared state/lists or SQL tables, or a historical environment for RPC calls.

Block-trigger replay

A block trigger (block: { every: N }) has no event stream. Replay fires one dry-run per block in [A, B] where block % every == 0, deduped by block number, with trigger.args = { block_number, network }. With an explicit --to-block it needs no chain access (block numbers are computed, not fetched):

rflow replay every-fifth-block --from-block 100 --to-block 120

replay 20260802213528773ba1 - 'every-fifth-block' blocks 100..=120 on local (dry-run)
 block | tx | run       | steps | would send
 100   | -  | succeeded | 1 ok  | ...
 105   | -  | succeeded | 1 ok  | ...
 110   | -  | succeeded | 1 ok  | ...
 115   | -  | succeeded | 1 ok  | ...
 120   | -  | succeeded | 1 ok  | ...
 
5 event(s) matched, 5 run(s), 0 would-be send(s)

Listing & pruning sessions — rflow replay ls / prune

Replay and test sessions leave their journal behind: namespaced rows in rflow.workflows / rflow.workflow_runs, a per-session rindexer schema (event replays), and a .rflow/runtime/replay-<id>/ directory. rflow replay ls shows them all:

rflow replay ls

 session              | workflow          | kind   | created             | runs | runtime dir
 20260802213528773ba1 | every-fifth-block | replay | 2026-08-02 21:35:28 | 5    | yes
 ac03575f             | settle-saga       | test   | 2026-08-02 21:34:45 | 3    | -

rflow replay prune deletes them: namespaced journal rows (one guarded transaction), the rindexer schema (DROP SCHEMA … CASCADE) and the runtime directory. A selector is required, and it only ever touches namespaced sessions: a live workflow, its runs and its analytics schema are never deleted:

rflow replay prune --older-than 7d      # sessions created > 7d ago
rflow replay prune --session <id>       # one session by id
rflow replay prune --all                # every replay/test session

pruned 6 session(s) - live workflows and their runs were not touched

The replay_<id>__<name> / test_<id>__<name> naming pattern is reserved for sessions: rflow validate rejects a live workflow named into it, and prune refuses to touch any session name that matches a workflow declared in rflow.yaml.

Two requirements to know about

--live — replay with real money

--live boots the relayer engine and sends real transactions for every historical match that passes its checks. It takes the project lock (a running rflow start shares wallets and nonces) and makes you type send, not y, at a red prompt:

!!! LIVE REPLAY !!!
every historical 'echo-deposit' event in blocks 40..=46 that passes its checks will
SEND A REAL TRANSACTION through your relayers.
this is NOT a rehearsal - money moves.
type 'send' to proceed (anything else aborts):

Use it for deliberate historical catch-up, not for testing.

rflow test — one run from a fixture

rflow test <workflow> --fixture <fixture.json>  # trigger payload for the workflow's OWN kind
rflow test <workflow> --event <fixture.json>    # event fixture (event workflows)
rflow test <workflow> --from-tx <tx-hash>       # trigger decoded from a real receipt
rflow test <workflow> --cron-at <ISO8601>       # cron workflows
rflow test <workflow> --read-output <out.json>  # read/query workflows

No indexer: one run is claimed under a namespaced name and driven to a terminal state in dry-run mode. --from-tx fetches the receipt and decodes the log that matches your trigger:

rflow test echo-deposit --from-tx 0xfe24bb08da25fb6403d9d2348a30703686d25a015e64c0bfa5763dd3ac98aa1c

dry-run of 'echo-deposit' - built-in sends are suppressed; commands execute unless dry_run: skip
 
test ea233669f4c445d88fefa2978fe5ec12 - 'echo-deposit' (dry-run, trigger tx 0xfe24bb08da25fb6…)
 step        | action           | outcome   | detail
 gate (#1)   | read             | succeeded | "1000000000000000000000000"
 payout (#1) | send_transaction | succeeded | would send transfer(address,uint256) -> 0x5FbDB2315678afecb367f032d93F642f6…
 
test run succeeded (2 step rows journaled under 'test_ea233669__echo-deposit')

A fixture for every trigger kind — --fixture

--fixture <file> interprets the file against the workflow's own trigger kind and builds the exact trigger.* roots a live fire would have journaled. A fixture whose shape does not match the kind (a webhook body against an event workflow) errors. Each kind's shape:

rflow test deposit-hook --fixture webhook.json     # { "body": { "amount": "500" } }
rflow test every-fifth-block --fixture block.json  # { "block_number": 19000005 }

test 35d0890757294e008352ee15469b87bf - 'deposit-hook' (webhook dry-run)
 step            | action    | outcome   | detail
 log_amount (#1) | state_set | succeeded | would set state.webhook_amount = 500

Two convenience flags are sugar over --fixture:

rflow test nightly-report --cron-at 2026-08-02T09:00:00Z   # cron workflows
rflow test utilization-guard --read-output util.json       # read/query workflows: { "output": "…" }

--event is the event-only alias ({"args": {...}} plus optional envelope fields) kept for backwards compatibility; --fixture supersedes it for event workflows too.

Reads and simulations run against the configured RPC's current state because rflow does not pass a historical block argument. For one chosen chain snapshot, start a fork with anvil --fork-url "$ETH_RPC" --fork-block-number <n> and point the network's rpc: at it, optionally using a profile. --read-output supplies only the read/query trigger fixture; later read/query steps still execute. See historical-state limits.

Saga rehearsals — --with-waits

A wait_for: saga parks the run until a matching event or a timeout. In a rehearsal the wait settles immediately by default (would_wait_for). --with-waits <file> pins each wait step's outcome so you can rehearse both branches; it works on rflow test and rflow replay:

{
  "await_settle": { "event": { "args": { "id": "1", "payout": "990" } } }
}

rflow test settle-saga --fixture deposit.json --with-waits waits.json

The matched event becomes the wait step's output, so downstream steps run and can read steps.await_settle.output.*:

 step              | action    | outcome   | detail
 kickoff (#1)      | state_set | succeeded | would set state.saga_started = 1
 await_settle (#1) | wait_for  | succeeded | {"args":{"id":"1","payout":"990"}}
 finalize (#1)     | state_set | succeeded | would set state.saga_settled = 1

Swap the outcome to { "await_settle": { "timeout": true } } and the wait takes its on_timeout path (fail | continue | goto). Here on_timeout: fail dead-letters the run and the process exits non-zero:

 await_settle (#1) | wait_for  | failed    | timeout: wait_for timed out (fixture) before any condition matched

Wait steps not named in the file fall back to the default immediate settle. Fixtures key on the config step id: waits inside a foreach (fan[0], fan[1], ...) or a finally: block all resolve through their base id, so one "fan": {...} entry pins every item's outcome.

Version diff — --diff

--diff OTHER_PROJECT_DIR replays the same range/fixtures against the current project and another project's same-named workflow (both dry-run, isolated sessions), then prints the per-trigger delta (added / removed / changed would-sends and status changes):

rflow replay every-fifth-block --from-block 100 --to-block 120 --diff ../old-version

diff 'every-fifth-block' blocks 100..=120 - this project vs the other project
 trigger         | change    | before (this project) | after (other project)
 block:local:100 | unchanged | succeeded [no sends]  | succeeded [no sends]
 block:local:105 | removed   | succeeded [no sends]  | (did not fire)
 block:local:110 | unchanged | succeeded [no sends]  | succeeded [no sends]
 
5 matched, 0 changed, 0 added, 2 removed, 3 unchanged - behaviour DIFFERS over this history

The command exits non-zero when behaviour differs, so a diff can gate a PR. If the other project is missing the workflow or declares a different trigger kind, it errors. The diff compares dry-run decisions (status + would-send params, including the exact calldata; an args-only or multicall inner-call change counts as changed), not live outcomes.

CI output — --output json | junit

Both rflow test and rflow replay take --output human (default), json or junit. rflow test exits non-zero when any run failed, so a fixture doubles as a CI check; json/junit still print.

--output json emits one structured object ({ session, workflow, trigger_kind, runs: [ { trigger_key, status, steps } ], summary }). --output junit emits a <testsuite> with one <testcase> per run (failed / dead-lettered runs carry a <failure>) that GitHub Actions and GitLab render natively:

name: backtest
on: [pull_request]
jobs:
  fixtures:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: rflow }
        ports: ["5432:5432"]
    env:
      DATABASE_URL: postgresql://postgres:rflow@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v4
      - run: cargo install --git https://github.com/joshstevens19/rflow rflow_cli
      - name: rehearse the settle saga (both branches)
        run: |
          rflow test settle-saga --fixture fixtures/deposit.json \
            --with-waits fixtures/waits-event.json --output junit > settle.xml
      - name: publish results
        if: always()
        uses: mikepenz/action-junit-report@v4
        with:
          report_paths: "*.xml"

Non-goals

• No automatic historical-state reconstruction. Event/block replay supplies historical triggers, while later reads and simulations use the configured RPC state. rflow test --read-output supplies a read/query trigger's fixture; it does not stub later read: or query: steps. See Historical-state limits.
• No hosted simulation cluster. Replay runs locally against your Postgres.
• Not a Foundry replacement. rflow rehearses workflow decisions, not contract internals. Keep your Solidity tests.

Where this fits