Observability

rflow's single exposed port (config.port, default 3940) carries everything you need to watch a deployment: the health probe, a Prometheus /metrics endpoint, a built-in run-trace viewer and the read-only JSON API behind it.

History explorer — `GET /`

Open http://localhost:3940/ while rflow start is running. The run-trace viewer is a full history explorer, a single dependency-free HTML/CSS/JS file embedded in the binary (no build step, no external assets, works offline) with routed views:

Workflows overview — every workflow with pause state, trigger kind, last run / success / failure, success & failure counts, runs waiting, median / p95 duration, tx count, recent error kind and per-network cursor / head lag
Runs history — the full filter/search set (workflow, status, trigger, network, contract, relayer, tx hash, block, run/step id, error kind, approval status, session, time range), shareable via the URL hash
Run detail — the full step timeline: attempts, statuses, tx ids and hashes, redacted inputs/outputs, structured errors, durable-delay wake times, approval decisions, reorg linkage and the replay/test marker. A command step's parsed JSON output shows as steps.<id>.output (secrets redacted; the command's raw env is never journaled)
Transactions, Approvals, Waiting, Failures & dead-letters, Reorg responses, Replay/test sessions — one focused view each

Untrusted run data (event args, errors, HTTP bodies) is escaped to text, never rendered as HTML. When auth is on the shell shows a login card and sends a bearer on every API call. Full details: history explorer.

Each run records the config_version of the boot that claimed it. rflow runs show <id> prints it (next to the per-workflow config_hash resume fence) and it is on every run in the JSON /api, so a run is always traceable to the exact config that produced it. See versioning & config plan.

Prometheus metrics — `GET /metrics`

Standard text exposition format; point a Prometheus scrape job at it:

scrape_configs:
  - job_name: rflow
    static_configs:
      - targets: ["localhost:3940"]

Metric	Type	Labels	Meaning
`rflow_runs_total`	counter	`workflow`, `status`	Runs settled, by terminal status (`succeeded` / `failed` / `dead_letter` / `skipped`).
`rflow_run_duration_seconds`	histogram	`workflow`	Run wall time from first start to settle.
`rflow_steps_total`	counter	`workflow`, `step`, `status`	Step attempts settled in the journal (the `step` label is the journal step id; `foreach` iterations aggregate under one `<id>[]` series so fan-out size never becomes label cardinality).
`rflow_trigger_fires_total`	counter	`workflow`, `kind`	Trigger fires that claimed a run. Duplicate deliveries are excluded by the exactly-once gate.
`rflow_sends_total`	counter	`workflow`, `status`	Transaction lifecycle: `submitted` when handed to the relayer, then the terminal `succeeded` / `failed`.
`rflow_indexer_last_block`	gauge	`network`	Highest block any trigger cursor has processed.
`rflow_waiting`	gauge	`kind`	Runs parked waiting (`tx` / `delay` / `event` / `approval`), computed from the journal at scrape time.
`rflow_workflow_errors_total`	counter	`workflow`	Settled failures that fired at least one `workflow_error` watcher, labelled by the FAILING workflow.
`rflow_liveness_breaches_total`	counter	`workflow`	Liveness breaches opened (one per breach, not per tick).
`rflow_liveness_breached`	gauge	`workflow`	1 while a liveness breach is open, 0 once recovered.
`rflow_indexer_stalled`	gauge	`network`	1 while the chain head advances but every trigger cursor on the network is frozen past `monitoring.indexer_stall_after`.
`rflow_rpc_requests_total`	counter	`network`, `endpoint`	RPC requests through rflow's provider failover stack, attributed to the (credential-redacted) endpoint that served them.
`rflow_rpc_errors_total`	counter	`network`, `endpoint`	Transport-class RPC failures (timeout/connection). A revert is an answer, never an endpoint error.
`rflow_rpc_request_duration_seconds`	histogram	`network`, `endpoint`	RPC round-trip time through the failover stack.
`rflow_rpc_healthy`	gauge	`consumer`, `network`, `endpoint`	1 while an endpoint passes health checks (right chain, within `failover_after` errors and `max_lag` blocks), else 0. `consumer` distinguishes the three stacks: `reads` (rflow's own providers), `indexer` and `relayer` (the embedded engines).
`rflow_rpc_lag_blocks`	gauge	`consumer`, `network`, `endpoint`	Blocks an endpoint trails the best-known head of its network at the last probe round.
`rflow_rpc_endpoint_events_total`	counter	`consumer`, `network`, `kind`	RPC endpoint failover events (`kind` = `switched` / `degraded` / `recovered`), today fed by the embedded indexer's event stream; a `switched` event is a real traffic move.
`rflow_rpc_dead`	gauge	`network`	1 while the stall monitor cannot read the chain head from ANY configured endpoint and zero endpoints are healthy. Never conflated with "not stalled", and never raised by a one-off hiccup while a healthy fallback stands by.
`rflow_process_start_time_seconds` / `rflow_process_uptime_seconds`	gauge	—	Process basics.

Counters increment at the journal choke points: a series only moves after the corresponding database transition committed, so /metrics never claims something the journal doesn't show. Counters reset on restart (normal for Prometheus; use rate()/increase()).

JSON API

Stable, read-only endpoints. They are unauthenticated by default, or bearer-gated when config.server.auth.token is set (/health stays open for probes; /hooks/* keep their own HMAC):

Endpoint	Returns
`GET /live`	HTTP 200 while the HTTP server is running; independent of workflow/database health.
`GET /health`	`{status: "ok" \| "degraded", reasons: [...], project, workflows, runs: {<status>: n}, relayers, rpc}` — `degraded` (HTTP 503; healthy responses use 200) when the database is unreachable, a liveness breach is open, an indexer stall is detected or the chain head is unreadable from every rpc endpoint of a network; `reasons[]` explains each. `rpc` maps each network to its per-consumer failover health: `rpc.<network>.<consumer>` with consumer one of `reads` (rflow's own providers), `indexer`, `relayer`; each is `{endpoints: [{url (credential-redacted), healthy, active?, consecutive_failures?, last_block?, lag?, latency_ms?, error_rate?, last_error?}], last_failover?: {from, to, reason, at}}`.
`GET /api/runs?limit=50[&failed=true]`	Recent runs, newest first (limit capped at 500).
`GET /api/runs/:id`	One run with its full step journal and approvals.
`GET /api/workflows`	Every workflow with pause state, trigger kind, cursors and today's run count.
`GET /api/history/*`	The cursor-paginated, secret-redacted history explorer API: `runs`, `runs/:id/timeline`, `txs`, `approvals`, `waiting`, `reorgs`, `replays`, `workflows`, `export`.

curl -s localhost:3940/api/runs?limit=5 | jq '.runs[].status'

Watch mode — `rflow start --watch`

--watch hot-reloads rflow.yaml while rflow runs:

edits are debounced (500ms) and validated first: an invalid save prints the errors red and the running config keeps running untouched
a valid save drains the stack gracefully (in-flight runs journal their progress and resume, exactly like a restart) and boots the new config in-process.

Limitations: a reload restarts the embedded indexer/relayer engines in full, and .env is not re-read. Export changed environment variables and restart for those.

Preflight checks — `rflow validate --preflight`

Live connectivity checks before you deploy, printed as an ok/fail table (exit code 1 on any failure):

every network RPC is called with eth_chainId and compared against the declared chain_id (catches copy-pasted RPC urls pointing at the wrong chain)
the database connection is opened and queried
notification channels: Telegram gets a real getMe; Slack/Discord webhooks get a TCP reachability check by default, and --preflight-notify posts an actual [rflow preflight] connectivity check message (webhooks cannot be verified without posting); PagerDuty/Opsgenie/Twilio get an API-host reachability check
every command step: the executable resolves (on PATH or as a project-relative path with the executable bit), any local script argument exists, and a known interpreter reports its version (dynamic ${{ }} invocations are reported as skipped)
the signer/relayer bootstrap is deliberately not preflighted: creating wallets is a stateful operation rflow start owns.

Nothing in the output prints secrets: bot tokens, webhook URLs and connection strings are redacted to their hosts.

Probe status codes

Use /live for restart probes and /health for readiness and operational alerts. /live remains 200 while the HTTP server is running, including during a workflow liveness breach. /health returns 200 when healthy and 503 with a JSON explanation when degraded. Both are public by default; config.server.health_public: false requires authentication for both. A waiting HA standby has no HTTP server until it takes leadership. See the production runbook for rollout, recovery and monitoring setup.

History explorer — GET /

Prometheus metrics — GET /metrics