Workflows

A workflow is when this happens, do these things, in order. Each entry under workflows: (keyed by name) has one trigger and a list of steps.

workflows:
  copy-trade:
    paused: false
    permissions:
      relayers: [payout]
      max_value_per_tx: 1 ether
    concurrency:
      group: "${{ trigger.args.sender }}"
      on_conflict: queue
    rate_limit: { max: 10, per: 1h }
    circuit_breaker:
      max_failures: 3
      window: 10m
      on_trip: [pause, notify: { channel: ops }]
    trigger:
      event: { contract: AnyPool, name: Swap, network: ethereum }
    steps:
      - id: gate
        read: { ... }
      - id: mirror
        send_transaction: { ... }
        retry: { max_attempts: 3 }
      - id: report
        notify: { ... }
    finally:
      - id: audit
        http_call: { url: "${AUDIT_URL}", body: { run: "${{ run.id }}", status: "${{ run.status }}" } }
    on_failure: dead_letter

Fields

Steps run in order

A step list is strictly sequential: 1 → 2 → 3. There is no DAG and no needs: — the default mental model stays simple. Any prior step's results are addressable from any later step: ${{ steps.gate.output }}, ${{ steps.mirror.tx.hash }}, ${{ steps.mirror.status }}.

Every step shares these common fields around its action:

The action itself is exactly one of: send_transaction · read · query · command · http_call · notify · delay · wait_for · choose · state_set / list_add / list_remove.

foreach — fan a step out

Run one step's action once per element of a collection expression:

- id: payout
  foreach: "${{ from_json(steps.plan.output.body).recipients }}"
  max_parallel: 4          # default 1 (strictly sequential)
  batch_size: 10           # optional — item becomes the batch (an array)
  send_transaction:
    network: ethereum
    relayer: payout
    contract: USDC
    function: "transfer(address,uint256)"
    args: ["${{ item.address }}", "${{ item.amount }}"]

• Each iteration sees ${{ item }} and ${{ item_index }}; if: is evaluated per iteration.
• Each iteration journals under <id>[<i>] with its own idempotency key — sends stay exactly-once per element, and a crash resumes exactly the interrupted iterations (settled ones never re-run).
• The parent step settles last with output = the array of iteration outputs (skipped iterations contribute null).
• Failure policy: any iteration failing after its retries fails the whole step. With max_parallel: 1 later items never start; with parallelism, in-flight iterations drain (never cancelled mid-send) but nothing new starts.
• wait_for runs inside a foreach: each item parks its own <id>[<i>] wait rows and resumes exactly once (placement rules); on_timeout: goto is rejected here.

strategy.matrix — one definition, many runs

strategy:
  matrix:
    network: [ethereum, base, arbitrum]
  fail_fast: false        # stop remaining combos after the first failure
  max_parallel: 2         # cap combos running concurrently

Each trigger fire expands into one run per combination; steps read the combination as ${{ matrix.network }}. Every combination extends the trigger key (...:matrix:network=base), so each is individually exactly-once — a crash mid-expansion resumes idempotently.

finally — always-run steps

finally: steps run after the run settles, success or failure — audit trails, lock releases, "run finished" pings. They see the full context plus run.status (the terminal status). A finally: failure is logged and journaled but does not change the run's outcome. wait_for may park inside finally: (the run's terminal status never changes while it waits — placement rules); on_timeout: goto is rejected here.

permissions

permissions:
  relayers: [payout]          # only these relayers may be used by this workflow
  max_value_per_tx: 1 ether   # cap on native value per send

concurrency

concurrency:
  group: "${{ trigger.args.sender }}"   # expression → serialization key
  on_conflict: queue                    # queue | skip | cancel_in_progress | replace

Runs whose group expression evaluates to the same key are serialized — per-trader, per-vault, per-anything. Runs in different groups execute concurrently up to config.max_concurrent_runs.

on_conflict decides what a new run does when its group's lane is busy:

Superseded runs are deliberate cancellations — they never count as circuit-breaker failures.

rate_limit

rate_limit: { max: 10, per: 1h }
rate_limit: { max: 10, per: 1h, durable: true }   # restart-proof

At most max run starts per per window. Claims beyond the limit stay queued (never dropped) and start when the window has room — combined with the exactly-once gate, nothing is lost, just deferred.

By default the window is in-process, per rflow instance (a sliding window): a restart, or an HA takeover, starts with a fresh window, so a burst right after a restart can exceed the configured rate.

Set durable: true to persist the window in Postgres (rflow.rate_limit_buckets). The durable limiter is a fixed window claimed with one atomic INSERT … ON CONFLICT statement: the first run in a window anchors it, the window releases per later, and concurrent runs racing the same bucket can never exceed max. A restart or takeover then cannot reset a live window.

budgets

Top-level budgets: cap cumulative spend inside a rolling window across every workflow, run and send that attaches them — the blast-radius cap when something upstream goes wrong.

budgets:
  treasury-daily:
    window: 24h
    max_native_value: 10 ether      # cap on cumulative native value in the window
    max_token_value:
      USDC: "1000000"               # raw token units (matches contracts.USDC)
    scope:
      relayers: [treasury]          # only sends from these relayers charge it
      networks: [ethereum]          # only sends on these networks charge it
 
workflows:
  sweep:
    budgets: [treasury-daily]       # this workflow's sends charge the budget

Before a send broadcasts (after every other check, right before the relayer call) it reserves its native value first, then a decodable ERC20 amount, against each attached in-scope budget. If a reservation would exceed the cap the send fails budget_exceeded and nothing broadcasts — racing runs cannot overshoot the budget. The reservation is a durable row keyed by the send's idempotency key, so it is exactly-once and crash-safe: a queued send settles as spent, a failed send is released (refunded), and the recovery pass reconciles orphaned reservations by the same key. Inspect live consumption with rflow budgets ls.

See Budgets for the full config surface, the ERC20 second-class charging rules, validation and how budgets compose with permissions.max_value_per_tx.

circuit_breaker

circuit_breaker:
  max_failures: 3
  window: 10m
  on_trip: [pause, notify: { channel: ops }]

After max_failures terminally failed runs inside window, the circuit trips: pause pauses the workflow, notify: alerts a channel (pagerduty channels receive it as critical). State machine, persisted in Postgres (rflow.circuit_state — restarts do not forget a tripped circuit):

• closed → failures are counted in the window
• open → after a full window with no new failures, the workflow is un-paused and one probe run is allowed through (half_open). Only the breaker's own pause is lifted: an operator's rflow workflow pause placed during the incident survives the auto-reset (resume it yourself when ready)
• half_open → exactly ONE run at a time is admitted for the whole workflow — concurrency.group lanes and matrix combos wait behind the probe. Probe fails: re-trips immediately; probe succeeds: closes and clears the window

Retries and the failure taxonomy

Every step failure carries a kind from a fixed taxonomy:

simulation_failed | reverted | insufficient_funds | rpc_timeout | rate_limited |
nonce_conflict | timeout | expired | dropped | data_unavailable | assert_failed |
gas_cap_exceeded

retry: applies to pre-queue and off-chain failures only — rpc_timeout, rate_limited, data_unavailable, gas_cap_exceeded. A reverted or failed onchain transaction is a step failure and is never blind-retried; once a tx is queued, gas bumping and rebroadcast are the embedded relayer's job.

retry:
  max_attempts: 3
  retry_if: "${{ error.kind in ['rpc_timeout', 'rate_limited'] }}"   # optional
  backoff: 10s                                                       # optional

retry_if evaluates against the error context (error.kind, error.message). Without it, any retryable kind retries up to max_attempts.

Failure handling

When a step fails terminally (non-retryable kind, or attempts exhausted), the run fails and on_failure decides its fate:

• dead_letter (default) — the run lands in the Postgres DLQ with its full journal; inspect with rflow runs list --failed, re-fire with rflow runs retry <id>
• drop — log and forget
• halt — pause the workflow itself; no further runs until resumed

finally: steps still run on every path.