feat(hooks): outbound webhooks — push signed lifecycle events to external endpoints

[teknium1]

Summary

Hermes can now push signed lifecycle events to external HTTP endpoints — the outbound mirror of the inbound webhook platform. External systems (CI, dashboards, other agents, another Hermes instance's inbound webhook) get told when Hermes does something instead of polling for it.

Infographic

Changes

• agent/outbound_webhooks.py (new): parses hooks.outbound: from config.yaml and registers notify-only callbacks on the existing plugin hook manager — any VALID_HOOKS event (on_session_end, subagent_stop, post_tool_call, ...) can be subscribed per target. Payload wire shape mirrors shell hooks' stdin JSON, plus delivery_id + timestamp.
• Delivery: fire-and-forget through a bounded queue + one daemon worker thread — a dead endpoint can never stall a tool call. Bounded retries (conn errors/5xx once with backoff; 4xx never). Queue-full drops with a warning.
• Signing: HMAC-SHA256 over the raw body, GitHub-style X-Hermes-Signature-256: sha256=<hex>. secret_env (env var name) preferred over inline secret. Unsigned targets flagged by hermes hooks list.
• Wiring: registered at the same three entry points as shell hooks (cli.py, hermes_cli/main.py, gateway/run.py), sharing one load_config() call. Idempotent across repeat calls. HERMES_SAFE_MODE=1 skips registration.
• agent/shell_hooks.py: outbound added to reserved sub-keys so the shell-hook parser doesn't warn on it.
• hermes_cli/hooks.py: hermes hooks list now shows outbound targets (url, events, matcher, timeout, signed/UNSIGNED).
• website/docs/user-guide/features/hooks.md: new "Outbound Webhooks" section — config schema, wire format, receiver-side verification snippet, delivery semantics.

Footprint

Zero new model tools, zero new subsystems — rides the existing hook bus (invoke_hook() call sites unchanged). Config-only feature; no new env vars for behavior (secret_env points at a secret, which is what .env is for).

Validation

Check	Result
tests/agent/test_outbound_webhooks.py (27 tests: parsing, matcher, HMAC, payload shape, idempotence, real-HTTP-server delivery, 4xx no-retry, conn-error swallow)	✅
Sibling suites (test_shell_hooks.py, test_hooks_cli.py, consent)	✅ 97/97
E2E: real load_config() → register_from_config() → invoke_hook() → live local HTTP server, HMAC verified receiver-side, matcher filtering, idempotent re-registration, shell-hook parser clean on outbound key	✅
ruff check on touched files	✅

Notify-only by design: callbacks always return None — outbound targets can never block a tool call or inject context, unlike shell hooks. E2E test asserts this (invoke_hook results empty).

Hardening pass (post-review, researched against industry webhook practice)

• Single delivery_id: the X-Hermes-Delivery header and the signed body's delivery_id were two different uuid4s — now one value, so receiver-side dedupe works as documented.
• Redirects never followed: urllib's default handler converts a redirected POST into a body-less GET (signed payload silently dropped). 3xx now logs a misconfiguration warning and counts as delivery failure.
• atexit flush: the delivery worker is a daemon thread; short-lived processes (hermes chat -q, cron) exited before on_session_end left the queue. Bounded 5s drain at interpreter exit.
• Docs: receiver replay-protection guidance (dedupe on delivery_id, timestamp freshness window — both live inside the HMAC-signed body) + redirect semantics.

Live validation

Scenario	Result
Real hermes chat -q session → live receiver: on_session_start → post_tool_call(terminal) → on_session_end, header/body delivery_id equal	✅
Isolated-HERMES_HOME E2E (17 checks): real load_config(), 5-target fan-out, HMAC verified receiver-side, secret_env resolution, matcher filtering, 5xx retried once, 4xx not retried, 302 not followed, dead endpoint returns in <1ms, malformed entries skipped, hermes hooks list signed flags, SAFE_MODE skip, idempotent re-registration	✅ 17/17
tests/agent/test_outbound_webhooks.py (31 tests; redirect + atexit tests sabotage-verified to fail without the fix)	✅
Sibling suites (shell hooks, hooks CLI) + ruff	✅

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 40f7f4b}

all good!

[julio-cloudvisor]

We have been running this PR in production for a couple of days and I wanted to leave real deployment feedback, because we think this is the right design and we would like to see it merge.

How we run it: cherry-picked the PR head (a39bfbd) onto our deployment base, which also carries a series of local patches. It applied with zero conflicts and zero hunk adaptations, which matches the claim in the description: one new module plus a few wiring points, genuinely additive. We enabled hooks.outbound on all 8 of our gateways, each pointing at a single small receiver service on the same host.

What works well in practice:

• The per-profile secret model composes nicely. We give every gateway its own HMAC secret via secret_env (populated from our secrets manager), and the receiver verifies X-Hermes-Signature-256 exactly like a GitHub webhook, then dedupes on X-Hermes-Delivery. Per-profile secrets matter in a fleet: one leaked secret can only forge that one profile's events, not the whole fleet's. Might be worth a sentence in the docs recommending per-instance secrets for multi-instance setups.
• Fire-and-forget holds up under real load. We have not seen a tool call or session stall from a slow or dead endpoint, which was our main worry before adopting.
• Keeping the payload shape identical to the shell-hook stdin wire made our receiver trivial to write.

One gap worth addressing before (or right after) merge: loop amplification. The description explicitly pitches instance-to-instance federation, one instance's outbound target being another instance's inbound webhook. That is also the configuration where a cycle is easy to create by accident: A's on_session_end wakes B, B's on_session_end wakes A, and every hop is a fresh, valid, signed event, so signature checking does not help. There is currently no cycle budget anywhere in the chain: no hop counter in the payload or headers, and no emission rate cap (the bounded queue bounds memory, not event rate; a tight loop stays well under queue capacity while burning sessions). We have been burned by an accidental self-trigger loop in a different part of our automation, cost us real money before a human noticed, so we now look for a deterministic backstop in anything event-driven. Two non-blocking suggestions:

1. Add a hop-count field, for example X-Hermes-Hop plus a hop field in the payload: when a session was itself started by an inbound webhook that carried a hop count, outbound deliveries for that session re-emit it incremented, and the sender drops deliveries beyond a small configurable max. Cheap, deterministic, and it caps any cycle regardless of topology.
2. Failing that, a short "federation" note in the hooks doc telling receiver authors to enforce their own rate caps and dedupe. We do this today (our receiver rate-caps per sender profile), but the failure mode is non-obvious enough that it deserves a warning next to the feature that invites it.

One smaller doc note from operating it: since delivery is fire-and-forget with bounded retries and queue-full drops (all reasonable choices), we treat events as accelerators and keep a slow reconciliation sweep as backstop for anything state-changing. A sentence in the docs saying "do not make an event the sole trigger for a critical state transition" would set the right expectation for new users.

Happy to share our receiver-side verification snippet if useful. Thanks for building this; it replaces a whole class of polling for us.