Backtesting — replay, test & dry-run

Would your workflow have done the right thing? rflow replay answers with real historical blocks: it re-indexes a range, fires your trigger exactly as live would, runs every step in dry-run mode — simulation, gas caps, policy checks, assert_sim, recheck all included — and reports what it would have sent. rflow test is the single-shot sibling for one fixture or one real transaction — for every trigger kind. Nothing broadcasts unless you explicitly go --live.

No hosted automation platform lets you do this. Your YAML is testable against the chain's actual history before a single wei moves.

At a glance:

rflow replay — a range of history

rflow replay <workflow> --from-block <n> [--to-block <n>] [--timeout <secs>] [--live] [--yes]

Replay drives event triggers (via a bounded re-index of the range) and block triggers (one dry-run per block where block % every == 0, deduped by block number). Cron/webhook/read/query workflows have no block-bounded history to replay from — use rflow test with a fixture for those.

Every output below is a real session from the repo's token-transfer-relay example: three historical deposits landed on the local chain (250, 25 and 500 RFT — the workflow's where: only matches deposits ≥ 100), then:

rflow replay echo-deposit --from-block 40 --to-block 46

dry-run replay of 'echo-deposit' from block 40 to 46 - nothing will be sent
 
replay 20260724034558fe323d - 'echo-deposit' blocks 40..=46 on local_anvil (dry-run)
 block | tx                 | run       | steps | would send
 43    | 0xfe24bb08da25fb6… | succeeded | 2 ok  | transfer(address,uint256) -> 0x5FbDB231567…, gas 51710
 45    | 0x75a03bb563b6a7c… | succeeded | 2 ok  | transfer(address,uint256) -> 0x5FbDB231567…, gas 51710
 
2 event(s) matched, 2 run(s), 2 would-be send(s)
session journal: rflow runs list (workflow 'replay_20260724034558fe323d__echo-deposit') - live history untouched

The 25 RFT deposit at block 44 is correctly absent — the where: filtered it, exactly as live would. Each row is one claimed run: the triggering event, the per-step outcomes (2 ok / failed at <step>), and the decoded would-be send with its simulated gas.

What dry-run means, precisely

Every check that can run without side effects still runs: templates, permissions, relayer policy caps, simulation, assert_sim, gas caps, recheck. But nothing leaves the process:

The isolation model

A replay boots an isolated session: a bounded indexer manifest for just this workflow and range, registered under a namespaced name (replay_<session>__<workflow>) with a replay:<session>: trigger-key prefix. Consequences, all deliberate:

• live dedupe is untouched — a live claim and a replay claim of the same event coexist
• a live rflow start never picks up replay runs, and the replay executor never picks up live runs
• the workflow's own run history stays clean — replay runs journal under the namespaced name
• trigger.throttle is dropped for the session (a rehearsal must not consume the live throttle window — replay shows every match), the trigger is forced to confirmed-at-depth-0 (historical blocks are final), and decoded fires are never offered to live parked wait_for sagas
• paused: is ignored — replaying a paused workflow is an explicit operator action

Block-trigger replay

A block trigger (block: { every: N }) has no event stream — replay instead fires one dry-run per block in [A, B] where block % every == 0, deduped by block number, building trigger.args = { block_number, network } for each. It needs no chain access when you pass an explicit --to-block (the block numbers are computed, not fetched):

rflow replay every-fifth-block --from-block 100 --to-block 120

replay 20260802213528773ba1 - 'every-fifth-block' blocks 100..=120 on local (dry-run)
 block | tx | run       | steps | would send
 100   | -  | succeeded | 1 ok  | ...
 105   | -  | succeeded | 1 ok  | ...
 110   | -  | succeeded | 1 ok  | ...
 115   | -  | succeeded | 1 ok  | ...
 120   | -  | succeeded | 1 ok  | ...
 
5 event(s) matched, 5 run(s), 0 would-be send(s)

Listing & pruning sessions — rflow replay ls / prune

Replay and test sessions leave their journal behind — namespaced rows in rflow.workflows / rflow.workflow_runs, a per-session rindexer schema (event replays), and a .rflow/runtime/replay-<id>/ directory. rflow replay ls shows them all:

rflow replay ls

 session              | workflow          | kind   | created             | runs | runtime dir
 20260802213528773ba1 | every-fifth-block | replay | 2026-08-02 21:35:28 | 5    | yes
 ac03575f             | settle-saga       | test   | 2026-08-02 21:34:45 | 3    | -

rflow replay prune deletes them — namespaced journal rows (one guarded transaction), the rindexer schema (DROP SCHEMA … CASCADE) and the runtime directory. It requires a selector so a bare prune can never nuke everything, and it only ever touches namespaced sessions — a live workflow, its runs and its analytics schema are never deleted:

rflow replay prune --older-than 7d      # sessions created > 7d ago
rflow replay prune --session <id>       # one session by id
rflow replay prune --all                # every replay/test session

pruned 6 session(s) - live workflows and their runs were not touched

The replay_<id>__<name> / test_<id>__<name> naming pattern is reserved for sessions: rflow validate rejects a live workflow named into it, and — belt-and-braces for journals created before that rule — prune cross-checks every discovered session against the workflows declared in rflow.yaml and refuses to touch a name that is a live config workflow.

Two requirements to know about

--live — replay with real money

--live boots the relayer engine and sends real transactions for every historical match that passes its checks. It takes the project lock (a running rflow start shares wallets and nonces) and makes you type send — not y — at a red prompt:

!!! LIVE REPLAY !!!
every historical 'echo-deposit' event in blocks 40..=46 that passes its checks will
SEND A REAL TRANSACTION through your relayers.
this is NOT a rehearsal - money moves.
type 'send' to proceed (anything else aborts):

Use it for deliberate historical catch-up ("process everything I missed last week, for real"), not for testing.

rflow test — one run from a fixture

rflow test <workflow> --fixture <fixture.json>  # trigger payload for the workflow's OWN kind
rflow test <workflow> --event <fixture.json>    # event fixture (event workflows)
rflow test <workflow> --from-tx <tx-hash>       # trigger decoded from a real receipt
rflow test <workflow> --cron-at <ISO8601>       # cron workflows
rflow test <workflow> --read-output <out.json>  # read/query workflows

No indexer at all: one run is claimed under a namespaced name and driven to a terminal state in dry-run mode. --from-tx fetches the receipt and decodes the log that matches your trigger — the fastest way to ask "what would this workflow have done with that transaction?":

rflow test echo-deposit --from-tx 0xfe24bb08da25fb6403d9d2348a30703686d25a015e64c0bfa5763dd3ac98aa1c

dry-run of 'echo-deposit' - nothing will be sent
 
test ea233669f4c445d88fefa2978fe5ec12 - 'echo-deposit' (dry-run, trigger tx 0xfe24bb08da25fb6…)
 step        | action           | outcome   | detail
 gate (#1)   | read             | succeeded | "1000000000000000000000000"
 payout (#1) | send_transaction | succeeded | would send transfer(address,uint256) -> 0x5FbDB2315678afecb367f032d93F642f6…
 
test run succeeded (2 step rows journaled under 'test_ea233669__echo-deposit')

A fixture for every trigger kind — --fixture

--fixture <file> interprets the file against the workflow's own trigger kind and builds the exact trigger.* roots a live fire would have journaled. A fixture whose shape does not match the kind (a webhook body against an event workflow) errors clearly. Each kind's shape:

rflow test deposit-hook --fixture webhook.json     # { "body": { "amount": "500" } }
rflow test every-fifth-block --fixture block.json  # { "block_number": 19000005 }

test 35d0890757294e008352ee15469b87bf - 'deposit-hook' (webhook dry-run)
 step            | action    | outcome   | detail
 log_amount (#1) | state_set | succeeded | would set state.webhook_amount = 500

Two convenience flags are sugar over --fixture:

rflow test nightly-report --cron-at 2026-08-02T09:00:00Z   # cron workflows
rflow test utilization-guard --read-output util.json       # read/query workflows: { "output": "…" }

--event is the event-only alias ({"args": {...}} plus optional envelope fields) kept for backwards compatibility; --fixture supersedes it for event workflows too.

Reads and simulations run against the network's current state (an eth_call cannot time-travel on a normal RPC) — for state-at-height fidelity, point the network's rpc: at an archive fork (e.g. anvil --fork-url $ETH_RPC --fork-block-number <n>) via a profile. Read/query replay uses the --read-output you supply, not reconstructed archive state — see non-goals.

Saga rehearsals — --with-waits

A wait_for: saga parks the run until a matching event or a timeout. In a rehearsal there is no live event to wait for, so by default the wait settles immediately (would_wait_for). --with-waits <file> pins each wait step's outcome deterministically so you can rehearse both branches — available on rflow test and rflow replay:

{
  "await_settle": { "event": { "args": { "id": "1", "payout": "990" } } }
}

rflow test settle-saga --fixture deposit.json --with-waits waits.json

The matched event becomes the wait step's output, so downstream steps run and can read steps.await_settle.output.*:

 step              | action    | outcome   | detail
 kickoff (#1)      | state_set | succeeded | would set state.saga_started = 1
 await_settle (#1) | wait_for  | succeeded | {"args":{"id":"1","payout":"990"}}
 finalize (#1)     | state_set | succeeded | would set state.saga_settled = 1

Swap the outcome to { "await_settle": { "timeout": true } } and the wait takes its on_timeout path (fail | continue | goto) — here on_timeout: fail dead-letters the run (and the process exits non-zero), exercising the timeout branch:

 await_settle (#1) | wait_for  | failed    | timeout: wait_for timed out (fixture) before any condition matched

Wait steps not named in the file fall back to the default immediate settle. Fixtures key on the config step id: waits inside a foreach (fan[0], fan[1], ...) or a finally: block all resolve through their base id, so one "fan": {...} entry pins every item's outcome.

Version diff — --diff

Did your change alter behaviour over real history? --diff OTHER_PROJECT_DIR replays the same range/fixtures against both the current project and another project's same-named workflow (both dry-run, isolated sessions), then prints the per-trigger delta — added / removed / changed would-sends and status changes:

rflow replay every-fifth-block --from-block 100 --to-block 120 --diff ../old-version

diff 'every-fifth-block' blocks 100..=120 - this project vs the other project
 trigger         | change    | before (this project) | after (other project)
 block:local:100 | unchanged | succeeded [no sends]  | succeeded [no sends]
 block:local:105 | removed   | succeeded [no sends]  | (did not fire)
 block:local:110 | unchanged | succeeded [no sends]  | succeeded [no sends]
 
5 matched, 0 changed, 0 added, 2 removed, 3 unchanged - behaviour DIFFERS over this history

The command exits non-zero when behaviour differs, so a diff can gate a PR. If the other project is missing the workflow or declares a different trigger kind, it errors. The diff compares dry-run decisions (status + would-send params, including the exact calldata — an args-only or multicall inner-call change counts as changed), not live outcomes.

CI output — --output json | junit

Both rflow test and rflow replay take --output human (default), json or junit. rflow test exits non-zero when any run failed, so a fixture doubles as a CI check; json/junit still print.

--output json emits one structured object ({ session, workflow, trigger_kind, runs: [ { trigger_key, status, steps } ], summary }). --output junit emits a <testsuite> with one <testcase> per run (failed / dead-lettered runs carry a <failure>) that GitHub Actions and GitLab render natively:

name: backtest
on: [pull_request]
jobs:
  fixtures:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: rflow }
        ports: ["5432:5432"]
    env:
      DATABASE_URL: postgresql://postgres:rflow@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v4
      - run: cargo install --git https://github.com/joshstevens19/rflow rflow_cli
      - name: rehearse the settle saga (both branches)
        run: |
          rflow test settle-saga --fixture fixtures/deposit.json \
            --with-waits fixtures/waits-event.json --output junit > settle.xml
      - name: publish results
        if: always()
        uses: mikepenz/action-junit-report@v4
        with:
          report_paths: "*.xml"

Non-goals

• No perfect historical off-chain reconstruction. Read/query replay uses the --read-output value you supply (or, for real fork state, point rpc: at an archive fork via rflow dev --fork); rflow does not reconstruct historical off-chain API responses.
• No hosted simulation cluster. Replay runs locally against your Postgres.
• Not a Foundry replacement. rflow rehearses workflow decisions, not contract internals — keep your Solidity tests.

Where this fits