Web trigger (HTTP poll)

Start a workflow from a web data source — protocol announcements, governance forum posts, security advisories, oracle incident pages, exchange listings, risk feeds, status pages, internal APIs. rflow polls an HTTP(S) URL on an interval, parses the body into items (RSS/Atom entries, JSON array elements, or the text of HTML nodes), and fires one run per new item — or one run per content change with changed: true.

workflows:
  security-advisory-watch:
    trigger:
      web:
        url: https://example.com/security/feed.xml
        every: 5m
        mode: rss
        where: "${{ 'critical' in (item.title | lower) }}"
        idempotency_key: "${{ item.link }}"
    steps:
      - notify:
          channel: pager
          message: "Security advisory: ${{ item.title }} ${{ item.link }}"

HTML change detection:

trigger:
  web:
    url: https://example.com/status
    every: 1m
    mode: html_text
    selector: ".incident"
    changed: true

Fields

Modes → items

A mislabeled feed is a loud error (mode: rss against an Atom document tells you to use mode: atom), as are an unparseable body, a JSON path that selects nothing, and a non-2xx status — every failure is logged with the workflow and the redacted URL, and retried on the next tick. No silent skips.

The item root

Steps in a web-triggered workflow see the fired item under item.* (as the spec examples read) and, like every trigger, under trigger.args.*; the poll metadata rides under trigger.web.* (url (redacted), mode, fingerprint, observed_at). A foreach: step's own item shadows the trigger's inside that step — the full item stays addressable as trigger.args.*.

where and idempotency_key evaluate over the item root only — any other root is a validation error.

New-item detection and dedupe

Every (filtered) item is fingerprinted — SHA-256 over a canonical serialization — and persisted in rflow.web_trigger_fingerprints. Each poll diffs against the store and claims one run per NEW fingerprint through the same idempotency gate webhooks and streams use, under web:{workflow}:{sha256(key)}:

• default key = the fingerprint — an edited item (title or summary touch-up) counts as new and fires again
• idempotency_key: "${{ item.link }}" keys on a stable id instead, so edits never re-fire — this is what "duplicate web items do not create duplicate runs when keyed" means
• the first successful poll on a fresh store records a baseline and fires nothing — booting on an established feed does not fire its whole history
• the store is capped per workflow (oldest-seen items age out only after they leave the feed), so an unbounded feed cannot grow it forever.

changed: true — content change detection

Instead of per-item diffing, the whole (filtered) item list is fingerprinted as one content blob; when it differs from the previous poll, ONE run fires with { count, items } as the item, plus trigger.web.previous_fingerprint. The first observation is a baseline; a revert (A→B→A) fires again, like trigger.read's changed mode. Combine with where so an irrelevant part of the page never fires: only matching items are fingerprinted.

Etiquette — be a polite citizen

A web poller is a guest on someone else's server:

• every fetch sends an rflow/<version> User-Agent, so source operators can identify (and contact, and rate-limit) the traffic
• respect robots.txt and the source's published rate limits. rflow does not fetch robots.txt for you — you choose the URL and cadence; choose ones the operator allows
• the every floor is 30s, and the right value is usually much larger — poll no faster than the content actually changes (a security feed updates hourly, not secondly); prefer feeds/APIs over HTML pages when offered
• defaults are conservative: 30s timeout, 2 MiB body cap, at most 5 redirects
• rflow never hammers on failure — a failed poll waits for the next tick, exactly like a successful one.

Redaction

Feed and API URLs often carry keys in their query string. Payloads and logs only ever carry scheme+host+path; request headers (where credentials belong) are never journaled or logged, and header values are redacted in config-version snapshots.

Rehearse offline

rflow test <workflow> --fixture <file> builds the trigger payload from a fixture that is one parsed item (bare, or wrapped as { "item": {...} }), so you can exercise the steps without touching the source.

See the security advisory watch use case for a full template, and the stream trigger for push-style (WebSocket) external signals.