Unattended ops — alerting when nobody is watching

Unattended automation must report two things on its own: "a run broke" and the harder one, "nothing is happening when something should be". rflow ships three journal-backed guarantees:

1. the workflow_error meta-trigger — run failures fire a workflow, so your alerting is itself a workflow
2. per-workflow liveness — alert when a workflow claims no run past max_silence
3. indexer stall detection — alert when the chain moves but the trigger cursors don't.

All three journal their state in Postgres, so alerts are bounded (one per incident, not one per tick) and survive restarts.

The workflow_error meta-trigger

A trigger whose source is other workflows' failures:

rflow_version: 1
name: ops-watchdog
 
config:
  port: 3940
  db_connection: ${DATABASE_URL}
 
workflows:
  on-any-failure:
    trigger: 
      workflow_error: 
        workflows: "*"          # one name, a list, or '*' for everything #
        on: [dead_letter]       # default; add `failed` for on_failure: drop|halt #
    steps:
      - id: page
        http_call:
          url: https://ops.example.com/rflow-failures
          body:
            workflow: "${{ trigger.failed.workflow }}"
            run: "${{ trigger.failed.run_id }}"
            step: "${{ trigger.failed.step_id }}"
            kind: "${{ trigger.failed.error.kind }}"
            message: "${{ trigger.failed.error.message }}"

The trigger context

Guarantees

• Exactly once per settled source run: the watcher claims wferr:{source_run_id} through the normal claim gate. A rflow runs retry that dead-letters the same run again re-fires nothing; matrix combinations are distinct runs and fire individually.
• Recursion guard: a workflow whose own trigger is workflow_error never fires other watchers ('*' never matches another watcher). A broken alerter logs loudly instead of starting an alert storm.
• Watchers fire on the engine settle path, live mode only: rflow replay / rflow test rehearsal runs never feed them, and paused watchers skip.

Honest limits

• A workflow can't watch itself, and rflow validate errors on unknown or empty workflows: entries (watcher-watching-watcher is a warning; the guard means it never fires).
• Two settle paths deliberately do not fire watchers: runs dead-lettered because the workflow definition changed under them, and on_conflict-superseded runs. Both are operator/config outcomes, not production failures.

Liveness — alert when a workflow goes silent

Failures page you; silence is what unattended deployments die of. Declare how often a workflow is expected to fire:

rflow_version: 1
name: deposit-mirror
 
config:
  port: 3940
  db_connection: ${DATABASE_URL}
 
networks:
  - name: base
    chain_id: 8453
    rpc: ${BASE_RPC}
 
contracts:
  Vault:
    abi: ./abis/vault.json
    network: base
    address: "0x0000000000000000000000000000000000000000"  # your vault
 
notifications:
  channels:
    ops:
      telegram:
        bot_token: ${TG_BOT_TOKEN}
        chat_id: ${TG_CHAT_ID}
 
workflows:
  deposit-mirror:
    liveness: 
      max_silence: 30m         # expected to claim a run at least this often 
      notify: [ops]            # notifications.channels entries 
    trigger:
      event: { contract: Vault, name: Deposit, network: base }
    steps:
      - id: mirror
        notify:
          channel: ops
          message: "deposit ${{ trigger.tx_hash }} mirrored"

• The signal is the most recent run claim (any status): silence means "nothing even fired", which catches dead trigger sources (RPC rot, an upstream contract migration, a paused feed) before failure counts would.
• One alert per breach: a rflow.liveness_events row opens with the breach (the dedupe gate), and a "recovered" notice goes out when a new run claims. A re-breach is a new row and a new alert.
• Boot anchor: silence is measured from the later of the last claim and the stack boot, so a stack that was down all weekend doesn't page the moment it boots; every workflow gets a full quiet window first.
• Paused workflows are exempt: pausing is an operator decision, not an outage. Pausing closes an open breach quietly.

rflow status prints the liveness table (last run, max_silence, OK/BREACHED); /health reports degraded with the breach in reasons[] while one is open; /metrics carries rflow_liveness_breached{workflow}.

Indexer stall detection

The whole-pipeline version of liveness: if the chain head advances while every event-trigger cursor on a network is frozen past a window, the indexing pipeline is stalled, even though no workflow "failed".

rflow_version: 1
name: deposit-mirror
 
config:
  port: 3940
  db_connection: ${DATABASE_URL}
  monitoring: 
    indexer_stall_after: 5m    # default 5m 
    notify: [ops] 
 
networks:
  - name: base
    chain_id: 8453
    rpc: ${BASE_RPC}
 
contracts:
  Vault:
    abi: ./abis/vault.json
    network: base
    address: "0x0000000000000000000000000000000000000000"  # your vault
 
notifications:
  channels:
    ops:
      telegram:
        bot_token: ${TG_BOT_TOKEN}
        chat_id: ${TG_CHAT_ID}
 
workflows:
  deposit-mirror:
    trigger:
      event: { contract: Vault, name: Deposit, network: base }
    steps:
      - id: mirror
        notify:
          channel: ops
          message: "deposit ${{ trigger.tx_hash }} mirrored"

Semantics chosen to avoid false pages:

• a cursor that is behind but moving (historical backfill) is never a stall
• an unreachable RPC never silences the stall check, but it isn't a stall either: a head unreadable from every endpoint (zero healthy) opens its own __rpc:<network> breach (same one-alert-per-breach + recovery-notice discipline, its own rflow_rpc_dead{network} gauge) while the stall verdict is left untouched. A one-off read failure with a healthy fallback standing by (a normal failover transition) opens nothing
• boot floors the comparison, so a fresh boot never instant-alerts.

Stalls journal under the synthetic subject __indexer:<network> in the same rflow.liveness_events table, degrade /health, raise rflow_indexer_stalled{network} and alert monitoring.notify channels once per stall.

Severity mapping

Liveness and stall alerts ride the same notification channels as everything else. For pager-style channels the severity says "unattended deployment went dark":

Slack/Discord/Telegram/webhook channels get the plain message; Twilio gets an SMS.

Honest notes

• The monitor loop runs on the leader instance only (the one holding the project lock) and evaluates every 60 seconds (plus 0-10s boot jitter). max_silence: 5s is honored, but detection lands on the next tick, up to ~60s later. The tick is the alerting resolution; there is no sub-minute paging.
• The liveness/stall monitor tasks are lazy: they only run when some workflow declares liveness: or config.monitoring is set. The boot banner prints a monitor: line when they start. One always-on exception: projects with networks run the RPC health persister + endpoint event consumer regardless; they observe and persist for rflow status//health and never page.
• Alert sending is best-effort with the journal as truth: if every channel errors, the breach row still exists (visible in /health, /metrics, rflow status) and the notified column records the delivery failure. rflow does not retry-spam a broken channel every tick.