Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Observability

rflow's single exposed port (config.port, default 3940) carries everything you need to watch a deployment: the health probe, a Prometheus /metrics endpoint, a built-in run-trace viewer and the read-only JSON API behind it.

History explorer — GET /

Open http://localhost:3940/ while rflow start is running. The run-trace viewer has grown into a full history explorer — a single dependency-free HTML/CSS/JS file embedded in the binary (no build step, no external assets, works offline) with routed views:

  • Workflows overview — every workflow with pause state, trigger kind, last run / success / failure, success & failure counts, runs waiting, median / p95 duration, tx count, recent error kind and per-network cursor / head lag
  • Runs history — the full filter/search set (workflow, status, trigger, network, contract, relayer, tx hash, block, run/step id, error kind, approval status, session, time range), shareable via the URL hash
  • Run detail — the full step timeline: attempts, statuses, tx ids and hashes, redacted inputs/outputs, structured errors, durable-delay wake times, approval decisions, reorg linkage and the replay/test marker. A command step's parsed JSON output shows as steps.<id>.output (secrets redacted; the command's raw env is never journaled)
  • Transactions, Approvals, Waiting, Failures & dead-letters, Reorg responses, Replay/test sessions — one focused view each

Untrusted run data (event args, errors, HTTP bodies) is escaped to text, never rendered as HTML. When auth is on the shell shows a login card and sends a bearer on every API call. Full details: history explorer.

Each run also records the config_version of the boot that claimed it — rflow runs show <id> prints it (next to the per-workflow config_hash resume fence), and it is on every run in the JSON /api. That is the audit trail: a run is always traceable to the exact config that produced it. See versioning & config plan.

Prometheus metrics — GET /metrics

Standard text exposition format; point a Prometheus scrape job at it:

scrape_configs:
  - job_name: rflow
    static_configs:
      - targets: ["localhost:3940"]
MetricTypeLabelsMeaning
rflow_runs_totalcounterworkflow, statusRuns settled, by terminal status (succeeded / failed / dead_letter / skipped).
rflow_run_duration_secondshistogramworkflowRun wall time from first start to settle.
rflow_steps_totalcounterworkflow, step, statusStep attempts settled in the journal (the step label is the journal step id; foreach iterations aggregate under one <id>[] series so fan-out size never becomes label cardinality).
rflow_trigger_fires_totalcounterworkflow, kindTrigger fires that claimed a run — duplicate deliveries are excluded by the exactly-once gate.
rflow_sends_totalcounterworkflow, statusTransaction lifecycle: submitted when handed to the relayer, then the terminal succeeded / failed.
rflow_indexer_last_blockgaugenetworkHighest block any trigger cursor has processed.
rflow_waitinggaugekindRuns parked waiting (tx / delay / event / approval), computed from the journal at scrape time.
rflow_workflow_errors_totalcounterworkflowSettled failures that fired at least one workflow_error watcher, labelled by the FAILING workflow.
rflow_liveness_breaches_totalcounterworkflowLiveness breaches opened (one per breach, not per tick).
rflow_liveness_breachedgaugeworkflow1 while a liveness breach is open, 0 once recovered.
rflow_indexer_stalledgaugenetwork1 while the chain head advances but every trigger cursor on the network is frozen past monitoring.indexer_stall_after.
rflow_process_start_time_seconds / rflow_process_uptime_secondsgauge—Process basics.

Counters increment at the journal choke points — a series only moves after the corresponding database transition committed, so /metrics never claims something the journal doesn't show. Counters reset on restart (normal for Prometheus; use rate()/increase()).

JSON API

Stable, read-only endpoints — unauthenticated by default, or bearer-gated when config.server.auth.token is set (/health stays open for probes; /hooks/* keep their own HMAC):

EndpointReturns
GET /health{status: "ok" | "degraded", reasons: [...], project, workflows, runs: {<status>: n}, relayers} — degraded (never an error status) when the database is unreachable, a liveness breach is open or an indexer stall is detected; reasons[] explains each.
GET /api/runs?limit=50[&failed=true]Recent runs, newest first (limit capped at 500).
GET /api/runs/:idOne run with its full step journal and approvals.
GET /api/workflowsEvery workflow with pause state, trigger kind, cursors and today's run count.
GET /api/history/*The cursor-paginated, secret-redacted history explorer API: runs, runs/:id/timeline, txs, approvals, waiting, reorgs, replays, workflows, export.
curl -s localhost:3940/api/runs?limit=5 | jq '.runs[].status'

Watch mode — rflow start --watch

--watch hot-reloads rflow.yaml while rflow runs:

  • edits are debounced (500ms) and validated first — an invalid save prints the errors red and the running config keeps running untouched
  • a valid save drains the stack gracefully (in-flight runs journal their progress and resume, exactly like a restart) and boots the new config in-process.

Limitations: a reload restarts the embedded indexer/relayer engines in full, and .env is not re-read — export changed environment variables and restart for those.

Preflight checks — rflow validate --preflight

Live connectivity checks before you deploy, printed as an ok/fail table (exit code 1 on any failure):

  • every network RPC is called with eth_chainId and compared against the declared chain_id (catches copy-pasted RPC urls pointing at the wrong chain)
  • the database connection is opened and queried
  • notification channels: Telegram gets a real getMe; Slack/Discord webhooks get a TCP reachability check by default — pass --preflight-notify to post an actual [rflow preflight] connectivity check message (webhooks cannot be verified without posting); PagerDuty/Opsgenie/Twilio get an API-host reachability check
  • every command step: the executable resolves (on PATH or as a project-relative path with the executable bit), any local script argument exists, and a known interpreter reports its version (dynamic ${{ }} invocations are reported as skipped)
  • the signer/relayer bootstrap is deliberately not preflighted — creating wallets is a stateful operation rflow start owns.

Nothing in the output prints secrets: bot tokens, webhook URLs and connection strings are redacted to their hosts.