Workflow error pager

Template: workflow-error-pager · category: monitoring · risk: monitor_only

The catch-all pager for an unattended rflow deployment: a workflow_error meta-trigger watching "*" — every other workflow in the project — and paging a pagerduty or telegram channel the moment a run settles dead_letter (and, by default, failed too). No signer, no relayer, no transactions.

When to use it

• the first thing to add to ANY production rflow project — failures should page you, not wait in rflow dead-letter ls
• route run failures to the on-call rotation (pagerduty) or an ops chat (telegram)
• alongside liveness:, which covers the silence failures this trigger cannot see

Generate it

# usually: add the pager to the project it should watch
rflow add workflow workflow-error-pager
 
# or as a standalone starting project
rflow new --template workflow-error-pager

Non-interactive (CI/agents):

rflow new --template workflow-error-pager --yes --output ./pager \
  --answer pager=telegram --answer page_on_failed=false

The generated YAML

# recipe: partial
workflows:
  workflow-error-pager:
    trigger:
      workflow_error:
        workflows: "*"
        on: [dead_letter, failed]
    steps:
      - id: page
        notify:
          channel: pager
          message: "rflow failure: ${{ trigger.failed.workflow }} run ${{ trigger.failed.run_id }} settled ${{ trigger.failed.status }} at step ${{ trigger.failed.step_id }} (attempt ${{ trigger.failed.attempt }}) - ${{ trigger.failed.error.kind }}: ${{ trigger.failed.error.message }}"
    on_failure: drop

Inputs

Required env vars

DATABASE_URL, plus one credential set depending on the pager answer: PAGERDUTY_ROUTING_KEY (an Events API v2 integration key) or TG_BOT_TOKEN/TG_CHAT_ID. The generated .env.example contains exactly the set you chose.

Safety notes

• Recursion-guarded by rflow itself: a workflow whose trigger is workflow_error never fires other watchers — a broken pager logs loudly instead of starting an alert storm. That is also why the template ends in on_failure: drop, not dead_letter: the pager must never park runs a human has to clean up.
• Watching "*" matches every non-watcher workflow in the same project, including ones you add later — no list to maintain.
• Replay/rflow test rehearsal runs never feed the trigger.

Run it locally

docker compose up -d     # postgres on localhost:5448
# fill .env (pagerduty routing key or telegram credentials)
rflow validate
rflow start

To see it fire, add any workflow that dead-letters (e.g. an http_call to a port nothing listens on) and trigger it — the page arrives with the failing workflow, step, attempt and error kind. fixtures/failed-run.json documents the exact trigger.failed.* shape the message renders against.

Production checklist

• pagerduty: use a dedicated Events API v2 integration per rflow deployment, so pages carry the right service
• decide page_on_failed: true pages for drop/halt workflows too; false pages only for parked (dead_letter) runs that need a human
• add liveness: to the workflows that must never go silent — the pager only sees runs that FAIL, not runs that never start
• rehearse the path once: force a dead-letter, watch the page, then rflow dead-letter requeue the run

Common modifications

• narrow the watch: workflows: [usdc-transfer-relay, treasury-sweep] instead of "*"
• POST to an incident webhook instead of (or before) the page: swap the notify: for an http_call: — the fixture shows the payload shape
• keep on: [dead_letter] only, and let drop workflows fail silently by design