feat(egress): doctor + audit + Anthropic native — follow-up to #30179 - #35149
Closed
Bartok9 wants to merge 13 commits into
Closed
feat(egress): doctor + audit + Anthropic native — follow-up to #30179#35149Bartok9 wants to merge 13 commits into
Bartok9 wants to merge 13 commits into
Conversation
Adds a TLS-intercepting egress proxy for remote terminal sandboxes (Docker
v1; Modal/SSH to follow). When enabled, the sandbox holds opaque proxy
tokens; iron-proxy swaps them for real provider API keys at the egress
boundary. Compromising the sandbox leaks tokens that only work from behind
the proxy.
Wraps ironsh/iron-proxy (Apache-2.0, Go binary). Same lazy-install pattern
as the recently merged Bitwarden Secrets Manager integration — pinned
version, SHA-256 verified download into ~/.hermes/bin/iron-proxy, no apt
or sudo required.
Disabled by default. Run `hermes egress setup` to mint tokens and
`hermes egress start` to launch. The Docker backend then automatically
mounts the CA, sets HTTPS_PROXY + CA-bundle env vars, and adds the
host-gateway hostmap.
New surfaces:
hermes egress install — download the pinned iron-proxy binary
hermes egress setup — interactive wizard (supports --from-bitwarden)
hermes egress start — spawn the managed proxy daemon
hermes egress stop — SIGTERM (+SIGKILL after 5s grace)
hermes egress status — binary + config + pid + listening + mappings
hermes egress disable — flip proxy.enabled = false
hermes egress config — print the path to the generated proxy.yaml
Optional Bitwarden integration: `--from-bitwarden` sources the real
upstream credentials from a BSM project at proxy startup, so rotating a
key in the Bitwarden web app propagates to sandboxes on the next proxy
start without touching .env.
Hermes-side scope (v1):
agent/proxy_sources/iron_proxy.py — install + CA + config + lifecycle
hermes_cli/proxy_cli.py — `hermes egress` subcommand tree
hermes_cli/config.py — "proxy:" section in DEFAULT_CONFIG
hermes_cli/main.py — argparse wiring (uses 'egress'
because 'proxy' is the existing
inbound OAuth reverse proxy)
tools/environments/docker.py — CA mount, HTTPS_PROXY, CA-bundle
env vars, --add-host wiring
Hermetic tests cover the full lifecycle: token mint, mapping discovery,
config + mappings I/O, install pipeline (HTTP + tar + checksum all mocked),
subprocess lifecycle (Popen mocked), Docker backend arg builder.
A live E2E test (gated on HERMES_RUN_E2E=1) downloads the real iron-proxy
binary, spawns it, routes a curl request through it against a local fake
upstream, and verifies the Authorization header was swapped from the proxy
token to the real secret value (and the proxy token did NOT leak through
to upstream).
Failures (binary missing, port collision, bad token) never block agent
startup — they emit a warning and continue. The Docker backend refuses to
start a sandbox when proxy.enabled=true but the daemon is dead, unless
proxy.enforce_on_docker is explicitly set to false.
Docs: website/docs/user-guide/egress/{index,iron-proxy}.md
Tests: tests/test_iron_proxy.py (35), tests/test_iron_proxy_e2e.py (1)
P0 — must-fix
- iron_proxy: emit default upstream_deny_cidrs (loopback, IMDS
169.254.0.0/16, RFC1918) when caller passes None. Honours the docs
promise that cloud-metadata IPs are refused regardless of allowlist.
- iron_proxy: bind 127.0.0.1 (+ docker0 bridge IP on Linux) instead of
INADDR_ANY (':9090'). LAN peers with a leaked sandbox token could
otherwise spend the operator's API quota against any allowlisted
upstream.
- ensure_ca_cert: write the CA private key via os.open(..., 0o600)
instead of shutil.copy2+os.chmod — closes the TOCTOU window where
the key existed under the default umask.
- discover_uncovered_providers + proxy.fail_on_uncovered_providers
config: refuse to start (when strict) if env vars for non-bearer
providers (Anthropic native x-api-key, AWS SigV4, Azure OpenAI,
etc.) are present. Surfaces a wizard warning in non-strict mode.
P1 — should-fix
- start_proxy: build a minimal subprocess env (PATH/HOME/locale +
only the env names referenced by mappings) instead of os.environ
.copy(). Strips proxy-recursion vars (HTTPS_PROXY etc.). Stops
the proxy's /proc/<pid>/environ from leaking every host secret
to same-uid local processes.
- start_proxy: optional Bitwarden refresh path
(refresh_secrets_from_bitwarden=True, bitwarden_config=...).
When credential_source=bitwarden, cmd_start wires it in — that's
what delivers the rotation guarantee the docs make.
- build_proxy_config: wire audit_log into the rendered yaml
(log.audit_path). Parameter was accepted but never used.
- ensure_audit_log: pre-create the audit log with 0o600 perms so
iron-proxy inherits tight permissions instead of relying on umask.
- Rename 'hermes proxy ...' → 'hermes egress ...' in user-facing
strings (docstring, RuntimeError messages, post-setup banner).
- start_proxy: open log file with 0o600 perms and close the parent
fd immediately after Popen — fixes the per-restart fd leak.
- DockerEnvironment: detect collisions between docker_env and the
egress-controlling env vars (HTTPS_PROXY, SSL_CERT_FILE, etc.).
When enforce_on_docker=true, fail loud rather than silently
inverting the isolation; when false, warn and let docker_env win.
- proxy_cli: merge_mappings preserves existing tokens on re-setup;
--rotate-tokens flag re-mints all of them. Stops re-running
`hermes egress setup` from invalidating tokens baked into
already-running sandboxes.
- proxy_cli: --from-bitwarden fail-loud on disabled BW config,
missing access token, or empty vault. Previously fell through to
the env path while still writing credential_source: bitwarden.
- docker.py: narrow `except Exception` → `except ImportError`;
iron_proxy._read_tunnel_port_from_config: same. Bare excepts
were masking real config-load bugs.
- start_proxy: write pidfile via os.open with O_NOFOLLOW + 0o600
+ st_uid check. Refuses to follow a pre-existing symlink at the
pidfile path.
- mint_proxy_token docstring: document the 128-bit suffix entropy
explicitly (sha256 truncated to 32 hex chars).
P2 — follow-up
- start_proxy: poll-with-timeout (100ms cadence on _port_listening)
instead of an unconditional 5s sleep. Saves several seconds per
Docker container create when enforce_on_docker=true.
- docker.py: apply enforce_on_docker semantics when CA file vanishes
between status.configured check and CA mount. Previously returned
empty args silently.
- docker.py: refuse to mount when mappings.json is empty/corrupt
(was indistinguishable from upstream outage from inside the
sandbox).
- install_iron_proxy: tarfile.extract(..., filter='data') to silence
the PEP 706 deprecation and opt into the 3.14+ default.
- _proxy_state_dir: chmod 0o700 unconditionally; add
_proxy_state_dir_ro() so read-only callers don't create the dir.
- stop_proxy: re-verify pid before SIGKILL via /proc/<pid>/stat
starttime AND _pid_alive. Prevents SIGKILL'ing a recycled pid.
- _pid_alive: tightened cmdline check — basename match on argv[0]
plus an in-process nonce env var ('iron-proxy' in cmdline matched
'tail iron-proxy.log' and editors with the log open).
- docker.py: NODE_OPTIONS=--use-openssl-ca so Node.js routes through
the OpenSSL CA store SSL_CERT_FILE controls, narrowing the
Python/curl-replace vs Node-add asymmetry waefrebeorn flagged.
P3 — polish
- proxy_cli: dest='egress_command' (was 'proxy_command' which
collided lexically with the inbound OAuth subparser).
- iron_proxy_version: cache by binary path — get_status is called
per Docker container create, version is constant per binary.
- Drop unused `import sys` from iron_proxy.
- proxy_cli: `is not None` check on --tunnel-port (was treating 0
as falsy and silently substituting the default).
- proxy_cli cmd_disable: use get_status().pid instead of reaching
into ip._read_pid() (stale pidfile from a crashed run would have
fired a spurious "still running" warning).
- Tests: replace hardcoded /tmp/ca.* paths with tmp_path-derived
fixtures so tests are hermetic across hosts.
CI
- Windows footguns scanner: os.kill(pid, 0) is now gated behind
platform.system() != 'Windows' with a windows-footgun: ok marker;
signal.SIGKILL falls back to SIGTERM on Windows via
getattr(signal, 'SIGKILL', signal.SIGTERM).
- docs MDX compilation: replace bare `<https://…>` URLs with
`[text](url)` syntax (MDX-jsx parser rejects the angle-bracket
form).
Tests
- 32 new tests covering default deny CIDRs, bind policy, audit log
wiring, subprocess env minimization, CA TOCTOU 0o600, state dir
0o700, empty-mappings refusal, CA-vanished refusal, docker_env
collision detection, token preservation/rotate, uncovered provider
detection, and the proxy_cli command handlers + argparse wiring.
- All 156 tests in test_iron_proxy + test_iron_proxy_cli +
test_docker_environment + test_config pass locally.
Acknowledged but not addressed in this revision
- E2E test for HTTPS CONNECT + TLS-MITM path: existing E2E exercises
plain HTTP; full MITM coverage needs separate CI infra (real iron-
proxy binary + curl with custom CA). Tracked as follow-up.
- Cosign-style supply-chain verification for the binary checksum:
upstream iron-proxy doesn't sign releases yet. Accepted pattern
(same as Bitwarden integration); tracked as follow-up.
- CA rotation CLI (`hermes egress rotate-ca`): scope-cut to a
follow-up.
Reviewers: @annguyenNous @waefrebeorn @GodsBoy @erhnysr
The bws helper's warnings list contains non-secret status messages
('rate limited', 'project not found', etc.), but CodeQL's taint
analyzer can't distinguish those from the secrets dict returned by
the same call. Log the count instead of the strings — the warnings
are still observable via 'hermes secrets bitwarden status'.
…ings
GodsBoy 2nd-round P1 (all 4 addressed):
- _detect_docker_bridge_ip: replace `ip.count('.') == 3` heuristic with
ipaddress.IPv4Address validation + reject unspecified/loopback/multicast/
reserved/link-local/global addresses. Hostile `ip` shim on PATH used to
be able to inject 0.0.0.0 here and re-open INADDR_ANY binding.
- cmd_setup credential_source preservation: re-running `hermes egress
setup` without --from-bitwarden no longer silently downgrades a previous
bitwarden config back to env. Require --no-bitwarden to switch
explicitly; otherwise preserve the existing mode and surface the
decision.
- fail_on_uncovered_providers docstring/default mismatch: docstring used
to claim default=True; behavior was default=False. Resolved by
truth-in-advertising — docstring now correctly states default=False —
AND splitting providers into a strict LLM-specific tier
(_LLM_SPECIFIC_NON_BEARER_PROVIDERS, used by start blocking) and a
generic uncovered tier (used by wizard warnings). Generic cloud creds
(AWS_*, GOOGLE_APPLICATION_CREDENTIALS) no longer trip refuse-start
for operators using terraform/gcloud alongside Hermes. New
discover_blocked_providers() returns the strict subset.
- start_proxy poll-loop must verify listening before pidfile:
previously fell through deadline-expired as success and wrote a
pidfile for a non-listening daemon. Refactored into a do-while
shape, require `listening=True` for success, kill the child + unlink
the pidfile on failure paths.
GodsBoy 2nd-round P2 (the worth-keeping subset):
- O_NOFOLLOW + 0o600 + st_uid check on iron-proxy.log open (symmetric
with the pidfile and audit-log paths the same PR hardens).
- pidfile O_EXCL: refactored pidfile-write into _write_pidfile_safely
which uses O_EXCL to detect concurrent starts. EEXIST with a live
pid means "another start in progress" — refuse with actionable
message; EEXIST with a dead pid means "stale crash" — unlink and
retry once. Discriminates rather than racing.
- _VERSION_CACHE: invalidate on install_iron_proxy success;
don't cache empty stdout (would poison `hermes egress status` for
the lifetime of the process if first probe hit a corrupt binary).
- ensure_audit_log now RAISES on OSError instead of swallowing it as
a warning. Previous behavior let the daemon create the file under
the default umask, exactly the world-readable scenario the helper
was built to prevent. cmd_setup catches the new RuntimeError and
surfaces "✗" with the actionable message.
- SIGINT/SIGTERM handler scoped around the start_proxy poll loop:
Ctrl-C while waiting for `hermes egress start` no longer leaks an
orphan daemon with the port bound. Handler kills the child +
unlinks the pidfile before re-raising.
- pidfile written IMMEDIATELY after Popen, BEFORE the listening
verification. Parent dying during the poll loop now leaves a
pidfile pointing at the orphan so the next `hermes egress stop` can
clean up. Failure paths in the poll loop explicitly unlink.
- _DEFAULT_UPSTREAM_DENY_CIDRS: add ::ffff:0:0/96 (IPv4-mapped IPv6 —
closes the v6-resolved IMDS bypass), 100.64.0.0/10 (CGNAT / cloud
overlays / K8s pod networks), 198.18.0.0/15 (RFC2544 benchmark).
- _NON_BEARER_PROVIDERS split into LLM-specific (Anthropic / Azure /
Gemini — block when strict) vs generic-cloud (AWS_*, GCP appdefault
— warn-only).
- docker.py except narrowing: load_config can raise yaml.YAMLError on
a malformed config.yaml, not just ImportError. Two callsites
(collision check + precedence resolution) now catch yaml.YAMLError
via a sentinel `import yaml` and fail-safe to enforced mode.
GodsBoy 2nd-round P3:
- _reset_for_tests: was a no-op claiming symmetry with bitwarden;
now actually clears _VERSION_CACHE and _proxy_nonce so in-process
callers (notebooks, pytest -p no:xdist) don't see state leakage.
- tests/test_iron_proxy_cli.py: replaced hardcoded Path("/tmp/...")
with hermes_home/-derived fixtures. Matches the same cleanup we
did for test_iron_proxy.py in the previous round.
- --rotate-tokens confirmation gate: when there are existing tokens,
prompt for "rotate" confirmation (skipped when stdin isn't a tty
so CI/scripted use still works) AND back up the mappings to a
timestamped sibling before overwriting. Surface a no-op note when
rotate is requested with no existing tokens.
stephenschoettler (runtime-boundary review):
- #1 BWS silent degrade at proxy start: when credential_source=bitwarden
but the BWS access token or project_id is missing OR the fetch
returns no values for mapped providers, raise instead of silently
falling back to host env. cmd_start also pre-checks at the wizard
layer for actionable error messages. Opt-in escape hatch via new
`proxy.allow_env_fallback: true` config for migration scenarios.
- #2 docker_env collision detection extended: `docker_env:
{OPENROUTER_API_KEY: sk-real}` in config.yaml with enforce_on_docker:
true now raises just like an HTTPS_PROXY collision would. The
collision check pulls mapped provider names from load_mappings() at
call time.
- #3 PID nonce persisted to disk: cross-CLI-invocation stale-pidfile
defense now works. start_proxy writes the nonce next to the pidfile
(sibling 0o600), stop_proxy reads it back via _read_persisted_nonce()
and uses it as a _pid_alive signal in the new process. Falls back
to argv0 basename matching when the file is missing (legacy install).
arshkumarsingh:
- #1 NODE_OPTIONS append-merge: egress dict no longer sets NODE_OPTIONS
directly (would clobber the operator's --max-old-space-size etc.).
Carry the egress flag in a sentinel key
_HERMES_EGRESS_NODE_OPTIONS_APPEND; DockerEnvironment merges into the
existing NODE_OPTIONS in env_args computation with de-duplication.
- #2 docs: structured per-request audit log is at audit.log, not
iron-proxy.log (the latter is daemon stdout/stderr). Diagram and
step-7 text corrected; both file roles are now documented separately.
Tests
- Added 12 new tests in test_iron_proxy.py covering bridge-IP rejection
(parametrized over 8 dangerous inputs), default deny-list adjacency
(IPv4-mapped-v6 + CGNAT), blocked-providers strict-subset property,
_pid_proc_starttime parser with paren-containing comm,
stop_proxy SIGKILL suppression on starttime drift, _reset_for_tests
clear behavior, iron_proxy_version don't-cache-empty, NODE_OPTIONS
sentinel verification, ensure_audit_log raise-on-OSError, and
persisted-nonce roundtrip.
- Added 1 new test in test_iron_proxy_cli.py covering cmd_start
BWS-token-missing fail-loud.
- All 100 tests in test_iron_proxy + test_iron_proxy_cli pass; all 78
tests in test_docker_environment + test_config still pass.
Acknowledged but not addressed:
- GodsBoy P3 dead-code `extra_env` kwarg: kept (removing is a breaking
change for any out-of-tree caller; the kwarg is documented and works).
- Residual risks GodsBoy called out: iron-proxy in-memory secret
zeroisation (Go-binary territory, out of scope); _PROXY_SUBPROCESS_ENV
_ALLOWLIST cosmetic gaps (RUST_LOG, GOMAXPROCS); follow-up.
internals reference Pre-v3 the egress docs were 175 lines covering the basics: quick start, slash commands, security model, failure modes. After three rounds of PR review we added a half-dozen new config knobs, two new flags, a strict/warn tier split for uncovered providers, persisted-nonce cross-process defense, audit-log + log-file separation, NODE_OPTIONS append-merge, docker_env collision detection, etc. — none of which the user-facing doc reflected. This commit closes that gap end-to-end: website/docs/user-guide/egress/iron-proxy.md (175 → 567 lines) - Configuration section expanded with every new knob: fail_on_uncovered_providers, allow_env_fallback, upstream_deny_cidrs. - Tables for default allowed hosts + default deny CIDRs. - Bind policy section (loopback + docker bridge, NOT 0.0.0.0) with the operator-facing "why can't I hit the proxy from my LAN" answer. - Uncovered providers section with the strict tier (Anthropic / Azure / Gemini — block when fail_on_uncovered_providers=true) vs warn tier (AWS, GCP appdefault — present on every dev laptop, never block). - Bitwarden integration expanded: rotation semantics, fail-loud at start, the allow_env_fallback escape hatch, --no-bitwarden flag, the preserve-existing-source rule on plain re-setup. - Slash commands section with --no-bitwarden, --rotate-tokens, and the token-rotation operator playbook (confirmation gate, backup file naming, restart-required caveat). - State directory layout table covering all 9 files we create + their modes. - Audit log vs daemon log distinction (the arshkumarsingh #2 fix that motivated the corrected diagram). - CA distribution into the sandbox: full table of injected env vars, the Python/curl REPLACE vs Node ADD asymmetry caveat with the NODE_OPTIONS=--use-openssl-ca mitigation. - docker_env collision detection: what gets blocked, what gets warned, the migration escape hatch. - PID + nonce defense section explaining how iron-proxy.nonce works cross-CLI and the SIGKILL-suppress-on-recycle path. - Security model expanded with the new defenses (IPv4-mapped-v6 IMDS bypass closure, env-var leakage prevention, LAN-peer-with-token-leak coverage). - Failure modes extended for every new refuse-start path. - Troubleshooting section (180 new lines) with grep-friendly error matchers for each common failure: BWS token missing, uncovered provider refused, port collision, slow bind, 403 from proxy, SSL verification errors inside the sandbox, 401 from upstreams, address- in-use orphan recovery, per-request audit log inspection. website/docs/getting-started/quickstart.md - One-paragraph mention of the egress proxy under "Sandboxed terminal" so operators discover the feature when they enable Docker isolation. website/docs/reference/cli-commands.md - Top-level command table now lists `hermes egress` alongside `hermes proxy` (different purpose, different direction — call it out). - New `## hermes egress` section with full subcommand syntax, common flows (first-time setup, switching credential source, rotating tokens, adding upstream), and diagnostic shortcuts. website/docs/reference/environment-variables.md - New "Egress proxy (sandbox-injected)" section documenting every env var the Docker backend injects: HERMES_EGRESS_PROXY, HERMES_PROXY_TOKEN_<NAME>, HTTPS_PROXY/HTTP_PROXY/NO_PROXY, REQUESTS_CA_BUNDLE/SSL_CERT_FILE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS, NODE_OPTIONS append-merge, HERMES_IRON_PROXY_NONCE. - Also fixes a stale layout issue with the Persistent Shell table that had two trailing rows getting orphaned in the v3 commit. website/docs/developer-guide/egress-internals.md (NEW, 363 lines) - Module layout map (which file owns what). - Full lifecycle walkthrough for install / setup / start / stop with the actual function calls in order. - "Security invariants" section enumerating every load-bearing property with the regression test name that guards it. These are the rules contributors must preserve when touching the module: - filesystem perms (0o700 dir, 0o600 secrets, O_NOFOLLOW everywhere) - subprocess env minimisation (no os.environ.copy) - bind policy (loopback + docker bridge, never 0.0.0.0) - default deny CIDR coverage - audit log fail-loud - bitwarden fail-loud - docker_env collision detection - PID recycling defense - token preservation on re-setup - credential_source preservation - Extension points: adding a bearer-token provider, adding a non-bearer provider, wiring iron-proxy into a non-Docker backend, subscribing to per-request audit events. - Testing recipe (hermetic + E2E + CLI smoke). website/sidebars.ts - New `developer-guide/egress-internals` entry under Developer Guide → Internals (alongside acp-internals, cron-internals, trajectory-format). Build verification - `cd website && npm install && npx docusaurus build` succeeds locally. - All three new pages render to static HTML in all three locales (en + zh-Hans + ko). - No new broken links or broken anchors introduced (pre-existing warnings on translation stubs are unrelated).
Single content conflict in hermes_cli/config.py — kept BOTH the paste_collapse_threshold knobs from main and the proxy section from this branch (they're independent additions to DEFAULT_CONFIG). All 187 tests in test_iron_proxy.py + test_iron_proxy_cli.py + test_config.py pass post-merge.
propagate handler exit codes
Live testing the full wizard against the real v0.39.0 binary
(downloaded + extracted via our own install_iron_proxy()) surfaced
three real bugs that the unit tests couldn't catch:
1. `proxy.http_listens` (plural) — NOT a field in v0.39's config struct.
Our code emitted both `http_listen` (string) and `http_listens`
(list) believing v0.39 accepts both forms. The binary actually
rejects with "field http_listens not found in type config.Proxy"
at YAML unmarshal time, so the daemon fails to start. Confirmed
via strings(1) audit of the v0.39 binary — only `http_listen` is
tagged.
2. `log.audit_path` — NOT a field in v0.39's config.Log struct. Same
class of error: "field audit_path not found in type config.Log".
Per-request audit-log records are not separable from server-level
logs at this binary version.
3. `metrics.listen` defaults to ":9090" — which is the SAME port as
our default `tunnel_port: 9090`. Result: every operator who runs
`hermes egress setup` followed by `hermes egress start` gets
"bind: address already in use" because the proxy listener and the
metrics listener fight for port 9090. We now explicitly pin
`metrics.listen: 127.0.0.1:0` to give it an ephemeral loopback
port that can never collide with tunnel_port regardless of what
operator sets.
Plus a fourth bug — pre-existing but surfaced by the egress live
test — that affects every Hermes subcommand:
4. `hermes_cli/main.py` calls `args.func(args)` at the bottom of
main() but discards the return value. Every subcommand handler
that returns a non-zero exit code (cmd_start refusing because
`fail_on_uncovered_providers=true`, cmd_setup refusing because
--from-bitwarden but BWS unreachable, etc.) was silently exiting 0.
Fix: capture the handler's return value and `sys.exit(rc)` when
it's a non-zero int. Other subcommands' contracts unchanged
because they either return 0/None or don't return at all.
Validation:
- 188/188 in test_iron_proxy.py + test_iron_proxy_cli.py +
test_config.py pass post-fix.
- 5333/5337 in tests/hermes_cli/ pass; the 4 unrelated failures
(test_managed_installs.py + test_update_hangup_protection.py)
are pre-existing on main, not touched by this PR.
- Manual wizard run end-to-end with the v0.39.0 binary in an
isolated HERMES_HOME:
* `egress install` — downloads + SHA-256 verifies + extracts
* `egress setup` — generates CA, mints tokens, writes
proxy.yaml that the binary now accepts (no http_listens,
no audit_path, metrics pinned to 127.0.0.1:0)
* `egress start` — daemon binds 127.0.0.1:9090, listens=yes
* `egress status` — shows pid + listening + mappings
* `egress stop` — clean shutdown, pidfile + nonce removed
* Idempotent re-start returns the running pid without spawning
* curl through the proxy with the openrouter token gets
forwarded; an attacker host gets HTTP 403 (allowlist works);
169.254.169.254 gets HTTP 403 (deny CIDR works)
* Refuse-start paths exit 1 with actionable messages:
- `fail_on_uncovered_providers=true` + ANTHROPIC_API_KEY set
- `credential_source=bitwarden` + BWS_ACCESS_TOKEN unset
* `--rotate-tokens` confirmation gate fires via pty:
typing 'cancel' aborts; typing 'rotate' proceeds and
creates a mappings.json.rotated-<timestamp> backup
Test updates:
- `test_default_bind_is_loopback_not_zero_zero` — asserts the
singular `http_listen` is loopback AND asserts `http_listens`
(plural) is NOT in the rendered yaml.
- `test_default_bind_uses_loopback_on_linux` — replaces
`test_default_bind_includes_docker_bridge_on_linux`. v0.39
only supports one bind per daemon process, so the docker bridge
augmentation is dropped from the rendered config; sandboxes
reach the daemon via host.docker.internal -> host-gateway
mapping, so loopback-only is functional.
- `test_metrics_listener_pinned_to_loopback_ephemeral` — new
regression test asserting `metrics.listen == "127.0.0.1:0"`.
- `test_audit_log_kwarg_does_not_inject_audit_path_v039` —
replaces `test_audit_log_path_lands_in_yaml`. audit_log kwarg
is still accepted for forward compatibility but does NOT emit
log.audit_path until upstream supports it.
…vior The previous docs round (906b1da) described the integration the way we wanted it to work — `http_listens` plural with a docker bridge bind, dedicated `audit.log` for per-request JSON records. Live testing against the real v0.39.0 binary in [905ce58] surfaced that neither field exists in v0.39's config schema, and the docs were making promises the daemon couldn't keep. This commit walks every claim in the docs back to what the binary actually does today, while keeping the upgrade path explicit so the docs stay coherent when the pinned `_IRON_PROXY_VERSION` bumps: website/docs/user-guide/egress/iron-proxy.md - Bind policy section: rewritten. Was "loopback + docker bridge IP on Linux"; now "loopback only" with an explicit explanation that v0.39 only supports one bind per daemon and that host.docker.internal -> host-gateway mapping is what sandboxes use to reach the loopback bind. - Bind policy section adds a note on the metrics-port pin that the previous round of docs didn't even mention. - State directory layout table: `audit.log` description rewritten to acknowledge it's a pre-created sentinel for future binary versions, NOT something the v0.39 daemon writes to. - New section "Logging on iron-proxy v0.39" replaces the old "Audit log vs daemon log" section. Explicitly tells operators the daemon log is the single source of truth for both audiences on v0.39, with the upgrade path called out. - Data-flow diagram step 7: rewritten to send per-request records to `iron-proxy.log` on v0.39 with cross-link to the new logging section. - Diagram caption updated. - Security-model "allowlisted-host exfiltration" line: "audit log captures" -> "daemon log captures". - Security-model "LAN peer leak" line: removed the docker-bridge claim. - Troubleshooting section's per-request-inspection recipes: rewritten to use `iron-proxy.log` and explain when the split stream will land. - Limitations list gets a new bullet calling out the single-bind + combined-log v0.39 constraints + the auto-upgrade posture. website/docs/developer-guide/egress-internals.md - Bind policy invariant: documents the singular `http_listen` v0.39 schema constraint + dead-code-until-upgrade status of the bridge-bind path. - New "Metrics port collision" invariant documenting why `metrics.listen: 127.0.0.1:0` is non-negotiable. - Audit log fail-loud invariant adds the v0.39 schema constraint note + the new `test_audit_log_kwarg_does_not_inject_audit_path_v039` regression test. - "Subscribing to per-request audit events" section updated to send watchers at `iron-proxy.log` for v0.39 with the upgrade pivot called out. website/docs/reference/cli-commands.md - Diagnostic shortcut for tailing the audit log: `tail audit.log | jq` -> `tail iron-proxy.log | jq` with the v0.39 note inline. Build verification: - `npx docusaurus build` succeeds across all three locales (en + zh-Hans + ko). - New `#logging-on-iron-proxy-v039` anchor lands in the rendered HTML and the in-page cross-references resolve. - No new broken anchors introduced (pre-existing warnings on unrelated zh-Hans pages are unchanged). - No leftover stale `#audit-log-vs-daemon-log` or `#http_listens` references anywhere on the egress pages.
alt-glitch
added
type/feature
This was referenced May 30, 2026
|
Really appreciate the One thing worth flagging for plugin authors building observability on top of Hermes: the The audit log schema already has the right field for this: Might be worth a note in the plugin authoring guide once this lands — something like "if you need to know which provider served a request, read |
Contributor
Author
Polish — improved description bullets• |
# Conflicts: # hermes_cli/main.py # tools/environments/docker.py
…llback gate, audit.log truth Addresses GodsBoy's May 30 follow-up review on PR NousResearch#30179. P0 — Linux sandboxes could not reach the proxy: - _default_http_listen now binds the docker bridge gateway on Linux (host.docker.internal resolves to the bridge gateway there; the old loopback-only bind was unreachable from containers). Loopback stays the default on Docker Desktop platforms; bridge-less Linux falls back to loopback with a warning. - Live-testing the fix against the real v0.39 binary surfaced a second latent bug the host-side E2E had masked: v0.39's http_listen does NOT terminate CONNECT — tunnel_listen does. HTTPS_PROXY traffic through http_listen got 400s from upstream. build_proxy_config now binds tunnel_listen (CONNECT/MITM) on tunnel_port and http_listen (plain-HTTP forwards) on tunnel_port+1; docker.py points HTTPS_PROXY at tunnel_port and HTTP_PROXY at tunnel_port+1. - Liveness probes (start_proxy poll loop, get_status) now probe the CONFIGURED bind host via _read_http_listen_from_config() instead of hardcoded loopback, which would have killed a healthy bridge-bound daemon as 'never came up'. P2 — allow_env_fallback dead on the partial-secret path: - The missing-secret branch in _build_proxy_subprocess_env now honors proxy.allow_env_fallback exactly as its own error message promises. - cmd_start refuses (or warns, with the fallback flag) when credential_source=bitwarden but secrets.bitwarden is disabled/missing — closing the silent-degrade-to-host-env hole. P2 — audit.log decoy on v0.39: - Wizard pre-create failure downgraded to a warning (file is non-load-bearing until the version bump); success line qualified as 'reserved'; user + dev docs stop telling operators to wire monitoring to the path today. P3 — metrics comment no longer claims 'hermes egress status' surfaces the ephemeral metrics port (it can't; :0 is random and unrecorded). Validation: 180 unit tests pass (9 new), gated E2E passes, and a live end-to-end run against the real binary verified CONNECT-MITM with Authorization swap on tunnel_listen, plain-HTTP swap on tunnel_port+1, 403 on non-allowlisted hosts, and bind-host-aware status probing. Docs gain a Linux firewall troubleshooting section (container→docker0 INPUT drops, e.g. ufw default-deny).
…search#30179 Three composable features on top of the iron-proxy egress firewall: 1. hermes egress doctor — 11-check end-to-end health check with --json/--check/--no-network, brew-doctor-style fix-it hints, and credential redaction in failure messages. 2. hermes egress audit — structured audit log viewer (tail -f / grep --since / stats / export json|csv) with first-time-host and >5%-403 anomaly detection. Pure functions iter_audit_log / aggregate_audit_stats / detect_audit_anomalies for testability. 3. Anthropic native (x-api-key) support via 'hermes egress setup --with-anthropic'. Adds a parallel secrets rule matching x-api-key, mints a TokenMapping for api.anthropic.com, and removes Anthropic from the uncovered/blocked sets when opted in. Off by default. TokenMapping gains an auth_header field (back-compatible default 'Authorization'). Tests: +50 (30 doctor, 11 audit, 9 anthropic) + 4 CLI dispatch tests. Existing iron-proxy suite unchanged (101 -> still pass). No new dependencies (stdlib + existing pyyaml/rich/openssl only).
Bartok9
force-pushed
the
feat/iron-proxy-doctor-audit-anthropic
branch
from
June 11, 2026 03:23
1f7d24a to
ccc1168
Compare
This was referenced Jun 17, 2026
teknium1
force-pushed
the
feat/iron-proxy
branch
3 times, most recently
from
June 26, 2026 08:11
cfa96dd to
6ff7e78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened by Bartok at Daniel's request, in response to Teknium's invitation for a security review of the iron-proxy egress work — Teknium asked Catalin (@catalinmpit), who had deployed Hermes on a hardened Hetzner VPS (Tailscale/UFW/Cloudflare/fail2ban), to look over PR #30179. This is a new, complementary PR, not a review of #30179.
This PR answers that by adding operability and closing one of the largest LLM-specific scope cuts in #30179 — without trading away security or ease of use. Every feature is opt-in or read-only, stdlib-only, and composes with #30179 rather than rewriting it.
Why this complements #30179
PR #30179 deliberately scoped out three things to keep the core landing tight:
status+ tail logs to know if egress was actually wired.x-api-key) was left as warn-only "uncovered" — the single biggest LLM-specific gap, since Anthropic doesn't useAuthorization: Bearer.This PR fills exactly those three gaps. It builds on
feat/iron-proxyas a follow-up.What lands
1.
hermes egress doctor— end-to-end health checkA single read-only command that runs 11 checks (binary, CA expiry, config parse, mappings/env, daemon liveness, listening socket, per-host reachability, token-swap correctness, uncovered providers, docker DNS, SSRF/IMDS guard) and prints
brew doctor-style fix-it hints. Flags:--json(stable{checks[], summary{}}, exit 1 on any fail),--check NAME(repeatable),--no-network(CI/hermetic). Credentials in failure messages are redacted to ≤4 chars.2.
hermes egress audit— structured audit viewertail [-n N] [-f],grep PATTERN --since,stats --since,export --format json|csv. Anomaly detection instatssurfaces first-time upstream hosts (catches a quiet DNS-rebind to a newly-allowlisted host) and hosts with >5% 403 rate.-fuses a 250ms polling loop (no new dependency) and survives log rotation.--sincetakes30m/2h/7d/1w,today, or ISO-8601.3. Anthropic native (
x-api-key) — opt-in via--with-anthropicEmits a parallel
secretsrule withmatch_headers: ["x-api-key"], mints aTokenMappingscoped toapi.anthropic.com, and removesANTHROPIC_API_KEYfrom the uncovered/blocked sets when opted in. Off by default — the key may be used via OpenRouter (Bearer) rather than the native endpoint, and we won't break that flow silently.TokenMappinggains anauth_headerfield with a back-compatible"Authorization"default;mappings.jsonround-trips it and old files load cleanly.New surfaces
run_doctor(),DoctorReport,DoctorCheck,DOCTOR_CHECK_NAMESiter_audit_log,aggregate_audit_stats,detect_audit_anomalies,parse_since,audit_log_pathdiscover_xapikey_mappings,auth_headeronTokenMappinghermes egress doctor,hermes egress audit {tail,grep,stats,export},hermes egress setup --with-anthropicwebsite/docs/.../iron-proxy.md,cli-commands.mdFailure modes considered
datetime,socket,ssl,urllib,concurrent.futures,csv,re) plus existingpyyaml/rich/openssl. CA expiry is read viaopenssl x509 -enddate, not thecryptographypackage.doctorandauditnever start/stop/rewrite anything. No auto-rotate, no auto-restart.run_doctorreturns a single clear "not supported on Windows" fail, mirroring_platform_asset_name._redact_secretcaps failure-message tokens at last-4-chars; there's a test asserting the proxy token never appears in a 403 token-swap failure detail.hermes proxyOAuth aggregator, did not bump_IRON_PROXY_VERSION, did not rewrite any feat(egress): iron-proxy credential-injection firewall for sandboxes #30179 file beyond additive edits.Validation
hermes egress doctor --no-networkexit codeCoverage gaps
(real ones, not aspirational)
doctor'sreachabilityandtoken-swapchecks are exercised only with a monkeypatched_https_head_via_proxy— there's no live-network integration test (by design; the existing E2E test is gated behind a marker and uses a real binary). The actual HTTP-proxy wiring of_https_head_via_proxy(ProxyHandler + unverified SSL context) is not asserted end-to-end here.audit tail -ffollow-loop is not unit-tested (it's an infinite poll loop); only the non-follow path is covered.docker-dnscheck has no test that actually runs Docker — it's skipped/mocked. Behavior on a real Linux host with--add-hostis unverified by CI.{_raw}but thestats/anomaly aggregations key onupstream_host/status, which would then be empty.Ambiguity flags
(decisions made without confirming)
auth_headeronTokenMappingrather than a separate parallel list — chosen so a single mapping set can mix Bearer + x-api-key andbuild_proxy_configstays a straight loop. This changes the dataclass signature (additive, defaulted) and themappings.jsonschema (additive key, back-compat read).with_anthropicpersisted inconfig.yamlunderproxy.with_anthropicsodoctorandstartagree on whether Anthropic is covered. New config key.brew doctor.token-swapsemantics: treats 401 (and 200/404) as "swap fired, reached upstream" and 403 as "proxy refused / rule broken." This is the documented contract but depends on the proxy returning 403 specifically on allowlist/rule failure.