From 4eb98fc5aec7d14f491789e08c2aae51768a9220 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 10 Aug 2026 05:24:50 +0000 Subject: [PATCH 1/3] Initial plan From 5d1e3fccd7cb39b99c4b48a09b331a0c359eab5d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 10 Aug 2026 05:40:01 +0000 Subject: [PATCH 2/3] Add B21/B22/B23 runner-doctor catalog entries and supporting docs Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> --- .github/agents/self-hosted-runner-doctor.md | 18 ++++++++++++++++ .../workflows/self-hosted-runner-doctor.md | 9 ++++++++ .../shared/self-hosted-failure-modes.md | 8 +++++++ docs/awf-config-spec.md | 12 +++++++++++ docs/usage.md | 21 +++++++++++++++++++ 5 files changed, 68 insertions(+) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 545c6b0bf..322c98661 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -90,6 +90,9 @@ Prefer the narrowest match. Examples: - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690) - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) +- `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) +- Strict-security workflow (no `--legacy-security`) appears unable to reach a GitHub Actions `services:` container port via `--enable-host-access` → B22 (not an AWF defect; `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; compiler-side gap deriving ports from `services:` tracked separately in github/gh-aw-firewall#7132) +- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -146,6 +149,12 @@ B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode. +B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. + +B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) — Strict-security (`--network-isolation`, no `--legacy-security`) workflows can appear unable to reach a GitHub Actions `services:` container port (for example Postgres 5432) via `--enable-host-access`. This is **not an AWF defect**: `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs plus a `host.docker.internal` hosts-file entry, not host iptables). The actual gap is in the gh-aw compiler, which only emits these flags in legacy-security mode with a hardcoded `80,443,` port list and never derives ports from `services:`. **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; a regression test was added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open, tracked separately in github/gh-aw-firewall#7132. + +B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. @@ -253,6 +262,9 @@ Establish these facts before matching a failure mode: | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | +| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /host/sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | +| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows appear unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | Not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs + `host.docker.internal` hosts-file entry, not host iptables). The gh-aw compiler is the actual gap: it only emits these flags in legacy-security mode with a hardcoded `80,443,` port list, never deriving ports from `services:`. | **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; regression test added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open — tracked separately in github/gh-aw-firewall#7132. | `awf --enable-host-access --allow-host-ports 5432 --allow-domains host.docker.internal -- psql -h host.docker.internal -p 5432 ...` in strict mode (no `--legacy-security`) — succeeds on documented AWF versions | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) | +| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | ## Category C — GHES / GHEC / `ghe.com` @@ -333,6 +345,10 @@ Establish these facts before matching a failure mode: | `503 TCP_TUNNEL:HIER_NONE` (server field `-:-`) on an allowlisted API host in network-isolation/topology mode, specifically after a Tailscale-up step | B17 (Tailscale policy-routing captures the default route, making host-specific DNS servers unreachable; fixed in github/gh-aw-firewall#6705 with `filterForNetworkIsolation()` stripping non-portable DNS before Squid config is generated) | | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | +| `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | +| `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | +| Strict-security workflow appears unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | B22 (not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; the gh-aw compiler gap that never derives ports from `services:` is tracked separately in github/gh-aw-firewall#7132) | +| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | ## Known unresolved items @@ -344,3 +360,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash +- B22 / github/gh-aw-firewall#7132 — gh-aw compiler still only emits `--enable-host-access`/`--allow-host-ports` in legacy-security mode with a hardcoded `80,443,` port list; it does not derive ports from `services:` for strict-mode workflows +- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index a35d1fabc..99fd588df 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -119,6 +119,9 @@ Prefer the narrowest match. Examples: - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) +- `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) +- Strict-security workflow (no `--legacy-security`) appears unable to reach a GitHub Actions `services:` container port via `--enable-host-access` → B22 (not an AWF defect; `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; compiler-side gap deriving ports from `services:` tracked separately in github/gh-aw-firewall#7132) +- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -175,6 +178,12 @@ B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode. +B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. + +B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) — Strict-security (`--network-isolation`, no `--legacy-security`) workflows can appear unable to reach a GitHub Actions `services:` container port (for example Postgres 5432) via `--enable-host-access`. This is **not an AWF defect**: `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs plus a `host.docker.internal` hosts-file entry, not host iptables). The actual gap is in the gh-aw compiler, which only emits these flags in legacy-security mode with a hardcoded `80,443,` port list and never derives ports from `services:`. **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; a regression test was added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open, tracked separately in github/gh-aw-firewall#7132. + +B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index bc4fcac65..687c7160c 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -61,6 +61,9 @@ Establish these facts before matching a failure mode: | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | +| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /host/sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | +| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows appear unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | Not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs + `host.docker.internal` hosts-file entry, not host iptables). The gh-aw compiler is the actual gap: it only emits these flags in legacy-security mode with a hardcoded `80,443,` port list, never deriving ports from `services:`. | **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; regression test added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open — tracked separately in github/gh-aw-firewall#7132. | `awf --enable-host-access --allow-host-ports 5432 --allow-domains host.docker.internal -- psql -h host.docker.internal -p 5432 ...` in strict mode (no `--legacy-security`) — succeeds on documented AWF versions | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) | +| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | ## Category C — GHES / GHEC / `ghe.com` @@ -143,6 +146,9 @@ Establish these facts before matching a failure mode: | Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner | B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686) | | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | | `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | +| `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | +| Strict-security workflow appears unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | B22 (not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; the gh-aw compiler gap that never derives ports from `services:` is tracked separately in github/gh-aw-firewall#7132) | +| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | ## Known unresolved items @@ -154,3 +160,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash +- B22 / github/gh-aw-firewall#7132 — gh-aw compiler still only emits `--enable-host-access`/`--allow-host-ports` in legacy-security mode with a hardcoded `80,443,` port list; it does not derive ports from `services:` for strict-mode workflows +- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream diff --git a/docs/awf-config-spec.md b/docs/awf-config-spec.md index 543c947a4..8757502f0 100644 --- a/docs/awf-config-spec.md +++ b/docs/awf-config-spec.md @@ -274,6 +274,18 @@ preview. Selecting `firecracker` never falls back to another runtime. When DinD is detected, AWF preserves the detected `DOCKER_HOST` value for the agent environment (including MCP servers) so DinD-aware tooling can reach the correct daemon without manual workflow env overrides. +`security.allowHostPorts` (`--allow-host-ports`) works standalone together with +`security.enableHostAccess` (`--enable-host-access`) in strict security mode +(the default, without `--legacy-security`). `applySecurityMode()` preserves +both flags whenever `security.networkIsolation` is enabled: host access is +served entirely through Squid port ACLs plus a `host.docker.internal` +hosts-file entry, not host-side iptables, so it has no dependency on +`--legacy-security`. This lets strict-mode workflows reach a GitHub Actions +`services:` container port (for example Postgres on 5432) without dropping +network isolation — see [docs/usage.md](usage.md) for a worked example. +`security.allowHostServicePorts` (`--allow-host-service-ports`), which relies +on host iptables, remains suppressed in strict mode. + The following CLI flag has no config-file equivalent by design: - `-e, --env ` — inject a single environment variable into diff --git a/docs/usage.md b/docs/usage.md index b6df1aad7..e5626ebcb 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -463,6 +463,27 @@ sudo awf \ > **Security Note:** A future update will change the default behavior to only allow ports 80 and 443 unless `--allow-host-ports` is specified. Explicitly set `--allow-host-ports` now to ensure consistent behavior across versions. +### Example: GitHub Actions `services:` Container in Strict Mode + +`--allow-host-ports` works standalone with `--enable-host-access` in strict +security mode (the default, without `--legacy-security`/`--network-isolation` +disabled) — host access is served through Squid port ACLs and the +`host.docker.internal` hosts-file entry, not host iptables, so it does not +require legacy security. This lets a strict-mode workflow reach a GitHub +Actions `services:` container (for example Postgres bound to port 5432 on the +runner) without disabling network isolation: + +```bash +# GitHub Actions services: postgres: +# ports: ["5432:5432"] + +awf \ + --enable-host-access \ + --allow-host-ports 5432 \ + --allow-domains host.docker.internal \ + -- psql -h host.docker.internal -p 5432 -U postgres -c 'select 1' +``` + ### CONNECT Method on Port 80 The firewall allows the HTTP CONNECT method on both ports 80 and 443. This is required because some HTTP clients (e.g., Node.js fetch) use the CONNECT method even for HTTP connections when going through a proxy. Domain ACLs remain the primary security control. From 18a804c1a41554db8595d68ff606ab6a9db09715 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 10 Aug 2026 15:02:56 +0000 Subject: [PATCH 3/3] Correct runner doctor strict topology guidance Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com> --- .github/agents/self-hosted-runner-doctor.md | 12 ++++++------ .../workflows/self-hosted-runner-doctor.lock.yml | 2 +- .github/workflows/self-hosted-runner-doctor.md | 4 ++-- .../workflows/shared/self-hosted-failure-modes.md | 8 ++++---- docs/awf-config-spec.md | 14 ++++++-------- docs/usage.md | 11 ++++++----- 6 files changed, 25 insertions(+), 26 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 322c98661..768930272 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -91,7 +91,7 @@ Prefer the narrowest match. Examples: - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) -- Strict-security workflow (no `--legacy-security`) appears unable to reach a GitHub Actions `services:` container port via `--enable-host-access` → B22 (not an AWF defect; `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; compiler-side gap deriving ports from `services:` tracked separately in github/gh-aw-firewall#7132) +- Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) - `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) @@ -151,7 +151,7 @@ B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-late B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. -B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) — Strict-security (`--network-isolation`, no `--legacy-security`) workflows can appear unable to reach a GitHub Actions `services:` container port (for example Postgres 5432) via `--enable-host-access`. This is **not an AWF defect**: `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs plus a `host.docker.internal` hosts-file entry, not host iptables). The actual gap is in the gh-aw compiler, which only emits these flags in legacy-security mode with a hardcoded `80,443,` port list and never derives ports from `services:`. **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; a regression test was added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open, tracked separately in github/gh-aw-firewall#7132. +B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. @@ -262,8 +262,8 @@ Establish these facts before matching a failure mode: | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | -| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /host/sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | -| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows appear unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | Not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs + `host.docker.internal` hosts-file entry, not host iptables). The gh-aw compiler is the actual gap: it only emits these flags in legacy-security mode with a hardcoded `80,443,` port list, never deriving ports from `services:`. | **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; regression test added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open — tracked separately in github/gh-aw-firewall#7132. | `awf --enable-host-access --allow-host-ports 5432 --allow-domains host.docker.internal -- psql -h host.docker.internal -p 5432 ...` in strict mode (no `--legacy-security`) — succeeds on documented AWF versions | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) | +| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | +| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | | B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | ## Category C — GHES / GHEC / `ghe.com` @@ -347,7 +347,7 @@ Establish these facts before matching a failure mode: | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | | `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | -| Strict-security workflow appears unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | B22 (not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; the gh-aw compiler gap that never derives ports from `services:` is tracked separately in github/gh-aw-firewall#7132) | +| Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | ## Known unresolved items @@ -360,5 +360,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash -- B22 / github/gh-aw-firewall#7132 — gh-aw compiler still only emits `--enable-host-access`/`--allow-host-ports` in legacy-security mode with a hardcoded `80,443,` port list; it does not derive ports from `services:` for strict-mode workflows +- B22 / github/gh-aw-firewall#7132 — strict topology lacks a direct raw-protocol `services:` route, and gh-aw does not derive service ports; compiler-emitted host-access flags alone are insufficient - B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream diff --git a/.github/workflows/self-hosted-runner-doctor.lock.yml b/.github/workflows/self-hosted-runner-doctor.lock.yml index 79dfc6351..fa9a63709 100644 --- a/.github/workflows/self-hosted-runner-doctor.lock.yml +++ b/.github/workflows/self-hosted-runner-doctor.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"52ccb0785d6ae87de3c792a3835d7c4978aa3a3622c1ea85d1f139ad3ef9934c","compiler_version":"v0.86.1","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.78"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"65a104e952bc43097d3044ea0319cd713cf21c07cdf341e3767fd2f9a14b2b95","compiler_version":"v0.86.1","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.78"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"8914f47b6c1bb8a802c7549f5ac1a81434b66403","version":"v0.86.1"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44","digest":"sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44@sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44","digest":"sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44@sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44","digest":"sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44@sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.8","digest":"sha256:38bbea36cdb46a3c9d04d1db05e672966f5239b431a2022eb35881688e5721d8","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.8@sha256:38bbea36cdb46a3c9d04d1db05e672966f5239b431a2022eb35881688e5721d8"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.8.0","digest":"sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520","pinned_image":"ghcr.io/github/github-mcp-server:v1.8.0@sha256:d5a18c04b92714c309eb46a2305087e91a4dbd80420f6e462656699f95093520"}]} # This file was automatically generated by gh-aw (v0.86.1). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 99fd588df..649a19632 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -120,7 +120,7 @@ Prefer the narrowest match. Examples: - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) -- Strict-security workflow (no `--legacy-security`) appears unable to reach a GitHub Actions `services:` container port via `--enable-host-access` → B22 (not an AWF defect; `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; compiler-side gap deriving ports from `services:` tracked separately in github/gh-aw-firewall#7132) +- Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) - `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) @@ -180,7 +180,7 @@ B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-late B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit ` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds. -B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) — Strict-security (`--network-isolation`, no `--legacy-security`) workflows can appear unable to reach a GitHub Actions `services:` container port (for example Postgres 5432) via `--enable-host-access`. This is **not an AWF defect**: `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs plus a `host.docker.internal` hosts-file entry, not host iptables). The actual gap is in the gh-aw compiler, which only emits these flags in legacy-security mode with a hardcoded `80,443,` port list and never derives ports from `services:`. **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; a regression test was added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open, tracked separately in github/gh-aw-firewall#7132. +B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 687c7160c..dc72d1771 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -61,8 +61,8 @@ Establish these facts before matching a failure mode: | B18 | Azure CLI, Azure DevOps CLI, and Azure/ADO MCP servers fail inside the AWF agent sandbox even after OIDC login completes in runner setup steps and relevant Azure/Microsoft domains are allowlisted; `~/.azure` config is invisible inside the sandbox, and `ADO_MCP_AUTH_TOKEN`/`AZURE_CONFIG_DIR` are not forwarded when `--enable-api-proxy` is active | `.azure` was not in the whitelisted `$HOME` tool subdirs (`home.toolSubdirs`) in the canonical sandbox mount policy — it was actually in `home.forbiddenSubdirs` — so Azure CLI config written by runner setup steps never reaches the sandbox; separately, `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` were not in the always-forwarded host env var list, so the MCP bridge could not authenticate | **Fixed in AWF (PR github/gh-aw-firewall#6690, merged 2026-07-28):** `.azure` added to `home.toolSubdirs` and removed from `home.forbiddenSubdirs` in the canonical mount policy (applies to both compose and sbx runtimes); `AZURE_CONFIG_DIR` and `ADO_MCP_AUTH_TOKEN` added to the always-forwarded env vars in `src/services/agent-environment/env-passthrough.ts`. **Authentication caveat:** #6690 deliberately scrubs Azure token cache files (`msal_token_cache*`, `accessTokens.json`, `service_principal_entries.json`) — a pre-AWF `az login` session is **not** inherited by the sandbox; only account metadata (tenant IDs, subscription list) is mounted. The agent must perform OIDC re-login inside the sandbox using a fresh writable `AZURE_CONFIG_DIR` (e.g. `az login --federated-token $ARM_OIDC_TOKEN --service-principal --username $ARM_CLIENT_ID --tenant $ARM_TENANT_ID`); see github/gh-aw-firewall#6686 for the working pattern. Upgrade AWF to version including github/gh-aw-firewall#6690 to enable this in-sandbox re-login. | Inside the agent sandbox: `ls -la ~/.azure` — presence confirms fix; `echo $AZURE_CONFIG_DIR $ADO_MCP_AUTH_TOKEN` — non-empty on patched AWF when set on the runner; `az account show` will still fail until in-sandbox OIDC re-login is performed | github/gh-aw-firewall#6686, github/gh-aw-firewall#6690 | | B19 | AWF fails on a primary error (e.g. topology-peer attach failure: `Failed to connect container "awmg-mcpg" to network "awf-net": ... No such container`), but the **last prominent diagnostic** consumers see is instead a large `[WARN] Could not fix squid log permissions: Error: Command failed with exit code 1: chmod -R a+rX ... Operation not permitted` dump from best-effort cleanup, obscuring the real root cause | `preserveDirectory()` in `src/artifact-preservation.ts` ran a direct host-side `chmod -R a+rX` on Squid-owned log directories during cleanup and logged the full `execa` error object at `warn` level whenever it failed with `EPERM`/`EACCES` — even though this is an expected, benign outcome on rootless runners. The existing Docker-based rootless permission repair (`fixArtifactPermissionsForRootless()`) already classified these as benign debug output, but the direct chmod path did not reuse that classifier. | **Fixed in AWF (PR github/gh-aw-firewall#6939, merged 2026-08-04)**: `preserveDirectory()` now reuses the same benign-permission-error classifier as `fixArtifactPermissionsForRootless()`, demoting expected `EPERM`/`EACCES` chmod cleanup failures to concise `debug` output while still surfacing unexpected cleanup failures as warnings. This preserves the primary startup error as the last prominent diagnostic. Upgrade AWF to include github/gh-aw-firewall#6939. | Trigger any primary startup failure on a rootless/self-hosted runner where Squid-owned log files exist; on unpatched AWF, cleanup logs a full `chmod ... Operation not permitted` `execa` error object after the primary failure; on patched AWF this is reduced to a debug-level note | github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939 | | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | -| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /host/sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | -| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows appear unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | Not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode (host access is served via Squid port ACLs + `host.docker.internal` hosts-file entry, not host iptables). The gh-aw compiler is the actual gap: it only emits these flags in legacy-security mode with a hardcoded `80,443,` port list, never deriving ports from `services:`. | **Confirmed and documented in AWF (PR github/gh-aw-firewall#7152, merged 2026-08-09)**, no code change needed: `docs/awf-config-spec.md` now states `security.allowHostPorts` works standalone with `security.enableHostAccess` in strict mode; `docs/usage.md` adds a worked `psql -h host.docker.internal -p 5432` example; regression test added in `security-mode.test.ts`. The gh-aw-side compiler fix (deriving ports from `services:`) remains open — tracked separately in github/gh-aw-firewall#7132. | `awf --enable-host-access --allow-host-ports 5432 --allow-domains host.docker.internal -- psql -h host.docker.internal -p 5432 ...` in strict mode (no `--legacy-security`) — succeeds on documented AWF versions | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152 (compiler-side gap still open: github/gh-aw-firewall#7132) | +| B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | +| B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | | B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | ## Category C — GHES / GHEC / `ghe.com` @@ -147,7 +147,7 @@ Establish these facts before matching a failure mode: | `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure | B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939) | | `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | -| Strict-security workflow appears unable to reach a GitHub Actions `services:` container port (e.g. Postgres 5432) via `--enable-host-access` | B22 (not an AWF defect — `applySecurityMode()` already preserves `--enable-host-access`/`--allow-host-ports` in strict mode; the gh-aw compiler gap that never derives ports from `services:` is tracked separately in github/gh-aw-firewall#7132) | +| Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | ## Known unresolved items @@ -160,5 +160,5 @@ Flag these explicitly instead of implying there is a complete fix: - C5 / #3937 — full `GH_HOST` leak fix still requires gh-aw changes - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash -- B22 / github/gh-aw-firewall#7132 — gh-aw compiler still only emits `--enable-host-access`/`--allow-host-ports` in legacy-security mode with a hardcoded `80,443,` port list; it does not derive ports from `services:` for strict-mode workflows +- B22 / github/gh-aw-firewall#7132 — strict topology lacks a direct raw-protocol `services:` route, and gh-aw does not derive service ports; compiler-emitted host-access flags alone are insufficient - B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream diff --git a/docs/awf-config-spec.md b/docs/awf-config-spec.md index 8757502f0..bce70db66 100644 --- a/docs/awf-config-spec.md +++ b/docs/awf-config-spec.md @@ -274,15 +274,13 @@ preview. Selecting `firecracker` never falls back to another runtime. When DinD is detected, AWF preserves the detected `DOCKER_HOST` value for the agent environment (including MCP servers) so DinD-aware tooling can reach the correct daemon without manual workflow env overrides. -`security.allowHostPorts` (`--allow-host-ports`) works standalone together with +`security.allowHostPorts` (`--allow-host-ports`) is accepted together with `security.enableHostAccess` (`--enable-host-access`) in strict security mode -(the default, without `--legacy-security`). `applySecurityMode()` preserves -both flags whenever `security.networkIsolation` is enabled: host access is -served entirely through Squid port ACLs plus a `host.docker.internal` -hosts-file entry, not host-side iptables, so it has no dependency on -`--legacy-security`. This lets strict-mode workflows reach a GitHub Actions -`services:` container port (for example Postgres on 5432) without dropping -network isolation — see [docs/usage.md](usage.md) for a worked example. +(the default, without `--legacy-security`), but it does not provide a direct +route to raw-protocol GitHub Actions `services:` containers. Strict topology +intentionally omits the agent's `host.docker.internal` mapping and host-access +iptables bypass; use legacy security or a separately verified tunnel for direct +service clients. `security.allowHostServicePorts` (`--allow-host-service-ports`), which relies on host iptables, remains suppressed in strict mode. diff --git a/docs/usage.md b/docs/usage.md index e5626ebcb..99f9dafde 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -467,17 +467,18 @@ sudo awf \ `--allow-host-ports` works standalone with `--enable-host-access` in strict security mode (the default, without `--legacy-security`/`--network-isolation` -disabled) — host access is served through Squid port ACLs and the -`host.docker.internal` hosts-file entry, not host iptables, so it does not -require legacy security. This lets a strict-mode workflow reach a GitHub -Actions `services:` container (for example Postgres bound to port 5432 on the -runner) without disabling network isolation: +disabled). Direct raw-protocol access to a GitHub Actions `services:` container +is not available in strict (`--network-isolation`) mode: the isolated agent has +no `host.docker.internal` route, and clients such as `psql` cannot use Squid's +HTTP proxy protocol. Use legacy security or a separately verified tunnel. For +example, to reach a Postgres service container with legacy security: ```bash # GitHub Actions services: postgres: # ports: ["5432:5432"] awf \ + --legacy-security \ --enable-host-access \ --allow-host-ports 5432 \ --allow-domains host.docker.internal \