Unattended ops — alerting when nobody is watching

Automation you don't watch needs to tell you two things on its own: "a run broke" and — the harder one — "nothing is happening when something should be". rflow ships three journal-backed guarantees for that:

1. the workflow_error meta-trigger — run failures fire a workflow, so your alerting is itself a workflow
2. per-workflow liveness — alert when a workflow claims no run past max_silence
3. indexer stall detection — alert when the chain moves but the trigger cursors don't.

All three journal their state in Postgres, so alerts are bounded (one per incident, not one per tick) and survive restarts.

The workflow_error meta-trigger

A trigger whose source is other workflows' failures:

workflows:
  on-any-failure:
    trigger:
      workflow_error:
        workflows: "*"          # one name, a list, or '*' for everything
        on: [dead_letter]       # default; add `failed` for on_failure: drop|halt
    steps:
      - id: page
        http_call:
          url: https://ops.example.com/rflow-failures
          body:
            workflow: "${{ trigger.failed.workflow }}"
            run: "${{ trigger.failed.run_id }}"
            step: "${{ trigger.failed.step_id }}"
            kind: "${{ trigger.failed.error.kind }}"
            message: "${{ trigger.failed.error.message }}"

The trigger context

Guarantees

• Exactly once per settled source run: the watcher claims wferr:{source_run_id} through the normal claim gate. A rflow runs retry that dead-letters the same run again re-fires nothing; matrix combinations are distinct runs and fire individually.
• Recursion guard: a workflow whose own trigger is workflow_error never fires other watchers — a broken alerter logs loudly instead of starting an alert storm. ('*' therefore never matches another watcher.)
• Watchers fire on the engine settle path, live mode only — rflow replay / rflow test rehearsal runs never feed them, and paused watchers skip.

Honest limits

• A workflow can't watch itself, and rflow validate errors on unknown or empty workflows: entries (watcher-watching-watcher is a warning — the guard means it never fires).
• Two settle paths deliberately do not fire watchers: runs dead-lettered because the workflow definition changed under them, and on_conflict-superseded runs — both are deliberate operator/config outcomes, not production failures.

Liveness — alert when a workflow goes silent

Failures page you; silence is what unattended deployments die of. Declare how often a workflow is expected to fire:

workflows:
  deposit-mirror:
    liveness:
      max_silence: 30m         # expected to claim a run at least this often
      notify: [ops]            # notifications.channels entries
    trigger:
      event: { contract: Vault, name: Deposit, network: base }
    steps: [...]

• The signal is the most recent run claim (any status) — silence means "nothing even fired", which catches dead trigger sources (RPC rot, an upstream contract migration, a paused feed) long before failure counts would.
• One alert per breach: a rflow.liveness_events row opens with the breach (that row is the dedupe gate), and a "recovered" notice goes out when a new run claims. A re-breach is a new row and a new alert.
• Boot anchor: silence is measured from the later of the last claim and the stack boot — a stack that was down all weekend doesn't page you the moment it boots; every workflow gets a full quiet window first.
• Paused workflows are exempt — pausing is an operator decision, not an outage. Pausing closes an open breach quietly.

rflow status prints the liveness table (last run, max_silence, OK/BREACHED); /health reports degraded with the breach in reasons[] while one is open; /metrics carries rflow_liveness_breached{workflow}.

Indexer stall detection

The whole-pipeline version of liveness: if the chain head advances while every event-trigger cursor on a network is frozen past a window, the indexing pipeline is stalled — even though no workflow "failed".

config:
  monitoring:
    indexer_stall_after: 5m    # default 5m
    notify: [ops]

Semantics chosen to avoid false pages:

• a cursor that is behind but moving (historical backfill) is never a stall
• an unreachable RPC stays quiet here (the head can't be observed — that's a network problem, visible elsewhere)
• boot floors the comparison, so a fresh boot never instant-alerts.

Stalls journal under the synthetic subject __indexer:<network> in the same rflow.liveness_events table, degrade /health, raise rflow_indexer_stalled{network} and alert monitoring.notify channels once per stall.

Severity mapping

Liveness and stall alerts ride the same notification channels as everything else. For pager-style channels the severity says "unattended deployment went dark":

Slack/Discord/Telegram/webhook channels get the plain message; Twilio gets an SMS.

Honest notes

• The monitor loop runs on the leader instance only (the one holding the project lock) and evaluates every 60 seconds (plus 0–10s boot jitter). max_silence: 5s is honored, but detection lands on the next tick — up to ~60s later. Treat the tick as the alerting resolution; there is no sub-minute paging.
• Monitor tasks are lazy: nothing runs unless some workflow declares liveness: or config.monitoring is set. The boot banner prints a monitor: line when they start.
• Alert sending is best-effort with the journal as truth: if every channel errors, the breach row still exists (visible in /health, /metrics, rflow status) and the notified column records the delivery failure — rflow does not retry-spam a broken channel every tick.