diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index c729f23df..00c087122 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -93,7 +93,7 @@ Prefer the narrowest match. Examples: - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) - Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) -- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) +- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; AWF's chroot-side gap is fixed in github/gh-aw-firewall#7245; only the upstream installer/harness mismatch in github/gh-aw-firewall#7130 remains open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -109,6 +109,7 @@ Prefer the narrowest match. Examples: - `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS) - `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) - `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) +- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244) ### 4. Check for known gaps and notable fixes @@ -124,6 +125,8 @@ A18 / github/gh-aw-firewall#6684, github/gh-aw#48658 — Under `runner.topology: A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 — On ARC/DinD, `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle`. The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`), so in DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem rather than the runner's staged path, and writes made inside the container land somewhere the runner-side ingestion step never sees. **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05):** the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. +A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount. This caused either `awf-agent` startup failure (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`) or, once worked around, an entrypoint abort at `mkdir -p /host$HOME/.m2` under `set -e`. **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup. `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -154,7 +157,7 @@ B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM- B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. -B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. +B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. @@ -238,6 +241,7 @@ Establish these facts before matching a failure mode: | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | +| A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | ## Category B — Self-hosted runners @@ -265,7 +269,7 @@ Establish these facts before matching a failure mode: | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | | B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | | B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | -| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | +| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. | **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. | Confirm gh-aw took the cache-hit path (`GITHUB_PATH` already set). On AWF including github/gh-aw-firewall#7245, check entrypoint logs for `ensure_usr_local_bin_shims` / `prepare_usr_local_bin_overlay`; on older AWF, absence of host `/usr/local/bin/copilot` reproduces the ENOENT | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 | ## Category C — GHES / GHEC / `ghe.com` @@ -350,7 +354,9 @@ Establish these facts before matching a failure mode: | `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | | Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | -| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | +| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | +| `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | +| `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | ## Known unresolved items @@ -363,4 +369,4 @@ Flag these explicitly instead of implying there is a complete fix: - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash - B22 / github/gh-aw-firewall#7132 — strict topology lacks a direct raw-protocol `services:` route, and gh-aw does not derive service ports; compiler-emitted host-access flags alone are insufficient -- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream +- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream; AWF's own contribution to the symptom is fixed in github/gh-aw-firewall#7245 (chroot `/usr/local/bin/copilot` overlay), so this item now tracks only the gh-aw-side installer/harness mismatch diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 9c26f3a4f..02672bbcd 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -122,7 +122,7 @@ Prefer the narrowest match. Examples: - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) - `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) - Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports) -- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) +- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; AWF's chroot-side gap is fixed in github/gh-aw-firewall#7245; only the upstream installer/harness mismatch in github/gh-aw-firewall#7130 remains open) - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files) - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs) - `FATAL: http_port: IPv6 is not available` → B3 @@ -137,6 +137,7 @@ Prefer the narrowest match. Examples: - `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS) - `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658) - `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) +- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244) - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) ### 4. Check for known gaps and notable fixes @@ -153,6 +154,8 @@ A18 / github/gh-aw-firewall#6684, github/gh-aw#48658 — Under `runner.topology: A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 — On ARC/DinD, `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle`. The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`), so in DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem rather than the runner's staged path, and writes made inside the container land somewhere the runner-side ingestion step never sees. **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05):** the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (container target stays `/tmp/gh-aw`; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. +A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount. This caused either `awf-agent` startup failure (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`) or, once worked around, an entrypoint abort at `mkdir -p /host$HOME/.m2` under `set -e`. **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup. `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -183,7 +186,7 @@ B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM- B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. -B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. +B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 5c4732068..f976c312b 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -36,6 +36,7 @@ Establish these facts before matching a failure mode: | A17 | On ARC/DinD with `runner.topology: arc-dind`, `--image-tag build-tools=sha256:` throws `Error: invalid key 'build-tools'`; the sysroot-stage init container image cannot be digest-pinned | `IMAGE_DIGEST_KEYS` in `src/image-tag.ts` does not include `'build-tools'`; `buildSysrootStageService()` constructs the image ref as a template string, bypassing `buildRuntimeImageRef()` entirely | **Fixed in AWF (PR github/gh-aw-firewall#5986)**: `'build-tools'` is now in `IMAGE_DIGEST_KEYS`; `buildSysrootStageService()` accepts `ParsedImageTag` via `SysrootServiceParams` and calls `buildRuntimeImageRef()` instead of a hardcoded template string. Upgrade to the AWF version that includes github/gh-aw-firewall#5986. | `awf --image-tag build-tools=sha256:abc ...` — on a patched version the command succeeds and the sysroot-stage image ref includes the pinned digest (visible in generated compose YAML); on older AWF without the fix the command throws `Error: invalid key 'build-tools'` | github/gh-aw-firewall#5985, github/gh-aw-firewall#5986 | | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 | | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | +| A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | ## Category B — Self-hosted runners @@ -63,7 +64,7 @@ Establish these facts before matching a failure mode: | B20 | On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy: tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe and failing the workflow before the agent starts | `awf-net` is `internal: true` with no outbound route. Squid and api-proxy are already dual-homed onto the external bridge (`awf-ext`), but `cli-proxy` was left attached only to `awf-net` even though it sets `extra_hosts: {'host.docker.internal': 'host-gateway'}` to reach the external DIFC proxy. Without a route out, Docker's `host-gateway` falls back to the default bridge gateway (`172.17.0.1`), unreachable from the isolated network | **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted; keeping `cli-proxy` on `awf-ext` would grant general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so do not treat upgrading to #7066 as a resolution. | Inspect cli-proxy logs for `ENETUNREACH ... :18443`; check generated compose for cli-proxy `networks:` — `awf-net` present and `awf-ext` absent on current builds; confirm `--network-isolation` is set with an external `difcProxyHost` | github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 | | B21 | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent container; `/sys/fs/cgroup` exposes no `pids.max`/`pids.current`, `ulimit -u` reports unlimited | AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability, so JVM tools can't discover or size against the real ceiling | **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09)**: new `--pids-limit ` CLI flag (default 1000, matches prior behavior) with `container.pidsLimit` config-file support, plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. | Inside agent: `cat /sys/fs/cgroup/pids.max` — presence confirms fix; raise ceiling with `--pids-limit 4000` for concurrent JVM builds | github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 | | B22 | Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example, Postgres on 5432) via `--enable-host-access` / `--allow-host-ports` | Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass. A raw PostgreSQL client cannot use Squid's HTTP `CONNECT` protocol, so allowing the port does not create a direct service route. gh-aw also does not derive service ports when it emits AWF flags. | **Known unresolved:** preserving or compiler-emitting `--enable-host-access` / `--allow-host-ports` is insufficient. A supported strict-topology route for raw service protocols is required in AWF, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel. | In strict mode, `getent hosts host.docker.internal` is absent and `psql -h host.docker.internal -p 5432 ...` cannot connect; do not treat emitted host-access flags alone as a successful probe | github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 | -| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Not an AWF defect. gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start* — a workaround symlink must exist on the host before `awf` is invoked. | **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gets a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps PATH-based lookups, not hardcoded-absolute-path harness bugs. Durable fix belongs in gh-aw's installer/harness (tracked as github/gh-aw-firewall#7130, open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`. | `ls -la /usr/local/bin/copilot` on the host before running `awf` — absence combined with a tool-cache hit reproduces the ENOENT; confirm gh-aw's installer took the cache-hit path (`GITHUB_PATH` already set) | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 | +| B23 | Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (cache hit) | Two gaps contributed to the symptom. The still-open upstream gap is that gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to PATH and returns early on cache hits (skipping the wrapper install to `/usr/local/bin/copilot`), while the compiler-emitted harness (`copilot_harness.cjs`) always spawns that hardcoded absolute path. Before github/gh-aw-firewall#7245, AWF also mounted host `/usr`/`/usr/local` read-only without creating the missing hardcoded entry inside the chroot, so the harness failed unless the host symlink already existed. | **Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):** `containers/agent/entrypoint.sh` adds `resolve_chroot_binary_path()`, `ensure_usr_local_bin_shims()`, and `prepare_usr_local_bin_overlay()`, invoked after `copy_dind_runner_binary`. When `AWF_ENSURE_USR_LOCAL_BIN=copilot` is set (auto-set for Copilot runs in `tool-specific-environment.ts`), AWF resolves the real `copilot` binary from `$GITHUB_PATH`, `AWF_HOST_PATH`, staged bin dirs, or system dirs, and creates `/usr/local/bin/copilot` inside the chroot via a read-only symlink-farm overlay without modifying host `/usr/local/bin`. Upgrade AWF to include github/gh-aw-firewall#7245. **Older AWF only:** before invoking `awf`, use the host workaround `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot`; it is unnecessary on patched AWF. The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130. PR github/gh-aw-firewall#7151 documents the older behavior and workaround. | Confirm gh-aw took the cache-hit path (`GITHUB_PATH` already set). On AWF including github/gh-aw-firewall#7245, check entrypoint logs for `ensure_usr_local_bin_shims` / `prepare_usr_local_bin_overlay`; on older AWF, absence of host `/usr/local/bin/copilot` reproduces the ENOENT | github/gh-aw-firewall#7130, github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245 | ## Category C — GHES / GHEC / `ghe.com` @@ -149,7 +150,9 @@ Establish these facts before matching a failure mode: | `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode | B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`) | | `unable to create native thread` / `Cannot create worker GC thread` from concurrent JVM builds (javac, Android manifest merger) inside the AWF agent; `/sys/fs/cgroup` shows no `pids.max`/`pids.current` | B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`) | | Strict-security workflow cannot reach a GitHub Actions `services:` raw-protocol port (e.g. Postgres 5432) | B22 (known unresolved: strict topology needs an AWF direct service route and gh-aw must derive service ports; emitted host-access flags alone are insufficient) | -| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open) | +| `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | +| `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | +| `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | ## Known unresolved items @@ -162,4 +165,4 @@ Flag these explicitly instead of implying there is a complete fix: - C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency (root cause unresolved; tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911) - D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause (SIGABRT `StringBytes::Encode`); one-shot retry mitigates (~8% failure rate) but does not prevent the crash - B22 / github/gh-aw-firewall#7132 — strict topology lacks a direct raw-protocol `services:` route, and gh-aw does not derive service ports; compiler-emitted host-access flags alone are insufficient -- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream +- B23 / github/gh-aw-firewall#7130 — gh-aw's `install_copilot_cli.sh`/`copilot_harness.cjs` mismatch on tool-cache hits (cache-hit path skips the `/usr/local/bin/copilot` wrapper while the harness always spawns that hardcoded path) remains open upstream; AWF's own contribution to the symptom is fixed in github/gh-aw-firewall#7245 (chroot `/usr/local/bin/copilot` overlay), so this item now tracks only the gh-aw-side installer/harness mismatch diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 47eab1aec..092a32ba6 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -92,6 +92,21 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('`checkUnknownModelRejection` now allows `provider === \'copilot\' && model.toLowerCase() === \'auto\'` to pass pre-flight'); expect(content).toContain('github/gh-aw-firewall#6810, github/gh-aw-firewall#6811'); expect(content).toContain('| `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12'); + // A20 new failure mode (writable /host$HOME under arc-dind sysroot staging) + expect(content).toContain('| A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start'); + expect(content).toContain('`filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); + expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):**'); + expect(content).toContain('github/gh-aw-firewall#7239, github/gh-aw-firewall#7244'); + expect(content).toContain('| `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 |'); + expect(content).toContain('| `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 |'); + // B23 update: PR #7245 fixes the AWF-side gap + expect(content).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); + expect(content).toContain('`ensure_usr_local_bin_shims()`'); + expect(content).toContain('`prepare_usr_local_bin_overlay()`'); + expect(content).toContain('AWF_ENSURE_USR_LOCAL_BIN=copilot'); + expect(content).toContain('The upstream installer/harness mismatch remains open in github/gh-aw-firewall#7130.'); + expect(content).toContain('**Older AWF only:**'); + expect(content).not.toContain('Not an AWF defect.'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); @@ -101,11 +116,13 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); expect(source).toContain('- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11'); expect(source).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); + expect(source).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)'); expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.'); expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); expect(source).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); expect(source).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); + expect(source).toContain('A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); expect(portableAgent).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(portableAgent).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); expect(portableAgent).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); @@ -115,5 +132,10 @@ describe('self-hosted runner doctor workflow config', () => { expect(portableAgent).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); expect(portableAgent).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); expect(portableAgent).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); + expect(portableAgent).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)'); + expect(portableAgent).toContain('A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); + expect(source).toContain('B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245'); + expect(source).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); + expect(portableAgent).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); }); });