From 5c3136abe88dc39c3390d1fb21dda2dd0d559ba3 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 28 Aug 2026 21:46:08 +0000 Subject: [PATCH 1/3] Initial plan From ad0ce55e41f8356f5d9cff3e72d16ec4a36572b8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 28 Aug 2026 21:50:02 +0000 Subject: [PATCH 2/3] docs: add runner doctor A22 and B27 --- .github/agents/self-hosted-runner-doctor.md | 10 +++++++++ .../workflows/self-hosted-runner-doctor.md | 6 ++++++ .../shared/self-hosted-failure-modes.md | 4 ++++ ...self-hosted-runner-doctor-workflow.test.ts | 21 +++++++++++++++++++ 4 files changed, 41 insertions(+) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index a3eb0df2a..f5c172f66 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -114,6 +114,8 @@ Prefer the narrowest match. Examples: - `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) - `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244) - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) +- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) +- `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) ### 4. Check for known gaps and notable fixes @@ -133,6 +135,8 @@ A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.t A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. +A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `buildContainerSecurityHardening` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `cat /proc/self/status | grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -171,6 +175,8 @@ B25 / github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 — On native-root B26 / github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 — In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net` in ~350ms. Two independent changes created the failure: `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI running inside cli-proxy must follow the `Location` header itself; meanwhile cli-proxy is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066), so it had no route to `productionresultssa*.blob.core.windows.net`. **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is routed through Squid while cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is not added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. Probe: reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP. +B27 / github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 — Docker Compose refuses to start AWF containers with repeated warnings `a network with name awf-net exists but was not created for project "awf-"` on persistent self-hosted runners. `generateDockerCompose()` named the Docker network `awf-net` without `external: true` or a project-scoped/unique name, so a stale network left behind by a killed or timed-out previous AWF run could be treated as belonging to a different project and block startup. **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. Probe: `docker network ls | grep awf-net` before a run; a stale `awf-net` not associated with a live compose project reproduces the mode, and patched AWF reclaims it automatically before compose up. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. @@ -255,6 +261,7 @@ Establish these facts before matching a failure mode: | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2//merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | +| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `buildContainerSecurityHardening` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `cat /proc/self/status \| grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or `docker info \| grep -i cgroup` for an approximate signal; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | ## Category B — Self-hosted runners @@ -286,6 +293,7 @@ Establish these facts before matching a failure mode: | B24 | Repeated `EACCES` retries from the harness reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or other files under `${RUNNER_TEMP}/gh-aw`) with no root-cause diagnostic explaining the native-root fallback ownership mismatch; occurs when AWF is invoked as native root (no `sudo`, so no `SUDO_UID`) | AWF falls back to a default sandbox uid/gid (`1000:1000`) when it cannot recover the original host identity from `SUDO_UID`. Files under `${RUNNER_TEMP}/gh-aw` created by the root-run harness stay root-owned, so the sandbox identity cannot read them once they are mounted read-only into the agent container | **Fixed in AWF (PR github/gh-aw-firewall#7565, merged 2026-08-20):** `isNativeRootWithoutSudo()` (`src/host-identity.ts`) detects root execution without `SUDO_UID` and logs an explicit warning identifying the fallback sandbox identity. `repairRunnerTempGhAwOwnership()` (`src/config-writer.ts`) recursively `chown`s `${RUNNER_TEMP}/gh-aw` to the resolved sandbox uid/gid (via `chown -h -P -R`, never following symlinks) before Docker Compose generation and container launch. Upgrade AWF to include github/gh-aw-firewall#7565. | Run AWF as root directly (no `sudo`) and check for the warning `Host process is running as root with no SUDO_UID; AWF will use sandbox identity ...`; compare `stat -c '%U:%G' "$RUNNER_TEMP/gh-aw/mcp-config/mcp-servers.json"` before/after a run — it should show the sandbox uid:gid after the fix instead of `root` | github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 | | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | +| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | ## Category C — GHES / GHEC / `ghe.com` @@ -374,6 +382,8 @@ Establish these facts before matching a failure mode: | Job exits 0 with no agent output/writes on a native-root runner (no `sudo`), even after `${RUNNER_TEMP}/gh-aw` ownership is fixed | B25 (checkout/container-workdir left root-owned while agent runs as fallback sandbox uid; fixed in github/gh-aw-firewall#7599) | | `Host workspace is not writable by the sandbox identity (:): ` | B25 | | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) | +| `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | +| `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index f913f9a0b..b7cc65487 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -142,6 +142,8 @@ Prefer the narrowest match. Examples: - `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959) - `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244) - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) +- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) +- `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) ### 4. Check for known gaps and notable fixes @@ -162,6 +164,8 @@ A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.t A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. +A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `buildContainerSecurityHardening` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `cat /proc/self/status | grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. + B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`. @@ -198,6 +202,8 @@ B25 / github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 — On native-root B26 / github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 — In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net` in ~350ms. Two independent changes created the failure: `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI running inside cli-proxy must follow the `Location` header itself; meanwhile cli-proxy is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066), so it had no route to `productionresultssa*.blob.core.windows.net`. **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is routed through Squid while cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is not added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. Probe: reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP. +B27 / github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 — Docker Compose refuses to start AWF containers with repeated warnings `a network with name awf-net exists but was not created for project "awf-"` on persistent self-hosted runners. `generateDockerCompose()` named the Docker network `awf-net` without `external: true` or a project-scoped/unique name, so a stale network left behind by a killed or timed-out previous AWF run could be treated as belonging to a different project and block startup. **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. Probe: `docker network ls | grep awf-net` before a run; a stale `awf-net` not associated with a live compose project reproduces the mode, and patched AWF reclaims it automatically before compose up. + D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor. D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 4c1757f28..152d6ab42 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -38,6 +38,7 @@ Establish these facts before matching a failure mode: | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2//merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | +| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `buildContainerSecurityHardening` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `cat /proc/self/status \| grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or `docker info \| grep -i cgroup` for an approximate signal; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | ## Category B — Self-hosted runners @@ -69,6 +70,7 @@ Establish these facts before matching a failure mode: | B24 | Repeated `EACCES` retries from the harness reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or other files under `${RUNNER_TEMP}/gh-aw`) with no root-cause diagnostic explaining the native-root fallback ownership mismatch; occurs when AWF is invoked as native root (no `sudo`, so no `SUDO_UID`) | AWF falls back to a default sandbox uid/gid (`1000:1000`) when it cannot recover the original host identity from `SUDO_UID`. Files under `${RUNNER_TEMP}/gh-aw` created by the root-run harness stay root-owned, so the sandbox identity cannot read them once they are mounted read-only into the agent container | **Fixed in AWF (PR github/gh-aw-firewall#7565, merged 2026-08-20):** `isNativeRootWithoutSudo()` (`src/host-identity.ts`) detects root execution without `SUDO_UID` and logs an explicit warning identifying the fallback sandbox identity. `repairRunnerTempGhAwOwnership()` (`src/config-writer.ts`) recursively `chown`s `${RUNNER_TEMP}/gh-aw` to the resolved sandbox uid/gid (via `chown -h -P -R`, never following symlinks) before Docker Compose generation and container launch. Upgrade AWF to include github/gh-aw-firewall#7565. | Run AWF as root directly (no `sudo`) and check for the warning `Host process is running as root with no SUDO_UID; AWF will use sandbox identity ...`; compare `stat -c '%U:%G' "$RUNNER_TEMP/gh-aw/mcp-config/mcp-servers.json"` before/after a run — it should show the sandbox uid:gid after the fix instead of `root` | github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 | | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | +| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | ## Category C — GHES / GHEC / `ghe.com` @@ -158,6 +160,8 @@ Establish these facts before matching a failure mode: | Job exits 0 with no agent output/writes on a native-root runner (no `sudo`), even after `${RUNNER_TEMP}/gh-aw` ownership is fixed | B25 (checkout/container-workdir left root-owned while agent runs as fallback sandbox uid; fixed in github/gh-aw-firewall#7599) | | `Host workspace is not writable by the sandbox identity (:): ` | B25 | | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) | +| `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | +| `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 37572356a..5156e6e46 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -106,6 +106,12 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('`/tmp/awf-lib` helper staging'); expect(content).toContain('Startup now fails closed'); expect(content).toContain('| `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) | A21 |'); + // A22 new failure mode (cap_drop filtered for trimmed capability bounding sets) + expect(content).toContain('| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list'); + expect(content).toContain('`invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"`'); + expect(content).toContain('`CapBnd`'); + expect(content).toContain('github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795'); + expect(content).toContain('| `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22'); // B23 update: PR #7245 fixes the AWF-side gap expect(content).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); expect(content).toContain('`ensure_usr_local_bin_shims()`'); @@ -144,6 +150,12 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('github/gh-aw-mcpg#10350'); expect(content).toContain('github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635'); expect(content).toContain('| `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26'); + // B27 new failure mode (stale awf-net collision on persistent self-hosted runners) + expect(content).toContain('| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project'); + expect(content).toContain('`generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net`'); + expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):**'); + expect(content).toContain('github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817'); + expect(content).toContain('| `a network with name awf-net exists but was not created for project` | B27'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); @@ -155,6 +167,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); expect(source).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)'); expect(source).toContain('- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21'); + expect(source).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); + expect(source).toContain('- `a network with name awf-net exists but was not created for project` → B27'); expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.'); expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); @@ -162,6 +176,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); expect(source).toContain('A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); expect(source).toContain('A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system`'); + expect(source).toContain('A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF\'s compose `cap_drop` list'); + expect(source).toContain('B27 / github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 — Docker Compose refuses to start AWF containers with repeated warnings'); expect(portableAgent).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(portableAgent).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); expect(portableAgent).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); @@ -174,7 +190,11 @@ describe('self-hosted runner doctor workflow config', () => { expect(portableAgent).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)'); expect(portableAgent).toContain('A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); expect(portableAgent).toContain('- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21'); + expect(portableAgent).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); + expect(portableAgent).toContain('- `a network with name awf-net exists but was not created for project` → B27'); expect(portableAgent).toContain('A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system`'); + expect(portableAgent).toContain('A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF\'s compose `cap_drop` list'); + expect(portableAgent).toContain('B27 / github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 — Docker Compose refuses to start AWF containers with repeated warnings'); expect(source).toContain('B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151, github/gh-aw-firewall#7245'); expect(source).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); expect(portableAgent).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); @@ -189,6 +209,7 @@ describe('self-hosted runner doctor workflow config', () => { expect(playbook).toContain('- `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode → B26'); expect(playbook).toContain('B26 / github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 — In `--network-isolation` mode'); expect(playbook).toContain('github/gh-aw-mcpg#10350'); + expect(playbook).toContain('B27 / github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 — Docker Compose refuses to start AWF containers with repeated warnings'); } }); }); From 4508fa3c07cc274ffbdcdf9c2c2a2619772a1f06 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 28 Aug 2026 22:21:26 +0000 Subject: [PATCH 3/3] docs: fix A22/B27 root-cause and probe accuracy, regenerate lock --- .github/agents/self-hosted-runner-doctor.md | 6 +++--- .github/workflows/self-hosted-runner-doctor.lock.yml | 2 +- .github/workflows/self-hosted-runner-doctor.md | 2 +- .github/workflows/shared/self-hosted-failure-modes.md | 4 ++-- scripts/ci/self-hosted-runner-doctor-workflow.test.ts | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index f5c172f66..804566085 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -135,7 +135,7 @@ A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.t A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. -A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `buildContainerSecurityHardening` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `cat /proc/self/status | grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. +A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. @@ -261,7 +261,7 @@ Establish these facts before matching a failure mode: | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2//merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | -| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `buildContainerSecurityHardening` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `cat /proc/self/status \| grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or `docker info \| grep -i cgroup` for an approximate signal; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | +| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | ## Category B — Self-hosted runners @@ -293,7 +293,7 @@ Establish these facts before matching a failure mode: | B24 | Repeated `EACCES` retries from the harness reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or other files under `${RUNNER_TEMP}/gh-aw`) with no root-cause diagnostic explaining the native-root fallback ownership mismatch; occurs when AWF is invoked as native root (no `sudo`, so no `SUDO_UID`) | AWF falls back to a default sandbox uid/gid (`1000:1000`) when it cannot recover the original host identity from `SUDO_UID`. Files under `${RUNNER_TEMP}/gh-aw` created by the root-run harness stay root-owned, so the sandbox identity cannot read them once they are mounted read-only into the agent container | **Fixed in AWF (PR github/gh-aw-firewall#7565, merged 2026-08-20):** `isNativeRootWithoutSudo()` (`src/host-identity.ts`) detects root execution without `SUDO_UID` and logs an explicit warning identifying the fallback sandbox identity. `repairRunnerTempGhAwOwnership()` (`src/config-writer.ts`) recursively `chown`s `${RUNNER_TEMP}/gh-aw` to the resolved sandbox uid/gid (via `chown -h -P -R`, never following symlinks) before Docker Compose generation and container launch. Upgrade AWF to include github/gh-aw-firewall#7565. | Run AWF as root directly (no `sudo`) and check for the warning `Host process is running as root with no SUDO_UID; AWF will use sandbox identity ...`; compare `stat -c '%U:%G' "$RUNNER_TEMP/gh-aw/mcp-config/mcp-servers.json"` before/after a run — it should show the sandbox uid:gid after the fix instead of `root` | github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 | | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | -| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | +| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | ## Category C — GHES / GHEC / `ghe.com` diff --git a/.github/workflows/self-hosted-runner-doctor.lock.yml b/.github/workflows/self-hosted-runner-doctor.lock.yml index 4b7ea666e..9d0942e82 100644 --- a/.github/workflows/self-hosted-runner-doctor.lock.yml +++ b/.github/workflows/self-hosted-runner-doctor.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"97c1365681a319a9d67e2fafb5ff7d55062709466efcddadcfdf46e8bcffc0e2","compiler_version":"v0.87.8","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"ae17f9515773acc6ff7e438595a1d8bc2b83ec5dfd21ff559a0f86728379457f","compiler_version":"v0.87.8","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} # gh-aw-manifest: {"version":1,"secrets":["GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"1aa033c7bf25ac9428fe521065b90c30a7070c4e","version":"v0.87.8"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.7","digest":"sha256:40a1e30b1b8d70642d4292485146cd5af612730d7a6a2e12706ddd13df375059","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.7@sha256:40a1e30b1b8d70642d4292485146cd5af612730d7a6a2e12706ddd13df375059"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.7","digest":"sha256:4f209dd4cbc74d47a6c7379956143de293429d1b1b2fb2647776cdcbf65836a1","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.7@sha256:4f209dd4cbc74d47a6c7379956143de293429d1b1b2fb2647776cdcbf65836a1"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.7","digest":"sha256:fb362a08d4d2f0da6c036e3f5d3b2fd87931e857fec3ca4a241cd2f2b61131f9","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.7@sha256:fb362a08d4d2f0da6c036e3f5d3b2fd87931e857fec3ca4a241cd2f2b61131f9"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.12","digest":"sha256:92d5377b6bd32cd5b9306b2a553f7ef3549bccff9207e46f931e7249bc718713","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.12@sha256:92d5377b6bd32cd5b9306b2a553f7ef3549bccff9207e46f931e7249bc718713"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b","pinned_image":"ghcr.io/github/gh-aw-node@sha256:529d02eb970b1161aa25c593a9c3df57fdfad5a8add328cb3b6eccef66f3183b"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"github","tools":["get_commit","get_file_contents","get_latest_release","get_me","get_pull_request","get_pull_request_comments","get_pull_request_diff","get_pull_request_files","get_pull_request_review_comments","get_pull_request_reviews","get_pull_request_status","get_release_by_tag","get_tag","issue_read","list_branches","list_commits","list_issue_types","list_issues","list_pull_requests","list_releases","list_starred_repositories","list_tags","pull_request_read","search_code","search_issues","search_pull_requests","search_repositories"]},{"name":"safeoutputs","tools":["add_comment","create_issue","missing_data","missing_tool","noop"]}]} # This file was automatically generated by gh-aw (v0.87.8). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index b7cc65487..2d0c85eab 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -164,7 +164,7 @@ A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.t A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`; `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. -A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `buildContainerSecurityHardening` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `cat /proc/self/status | grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. +A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF's compose `cap_drop` list with `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts such as Talos Linux that trim capabilities from the container capability bounding set. `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` entries for the Squid and agent containers without filtering them against the host/daemon's effective bounding set, so Docker Compose validation could fail before containers started. **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent are omitted as safe no-ops. Upgrade AWF to include github/gh-aw-firewall#7795. Probe: `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE`; pre-fix compose fails with `invalid CapDrop`, post-fix compose starts normally. B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`. diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 152d6ab42..2594ed600 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -38,7 +38,7 @@ Establish these facts before matching a failure mode: | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 | | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 | | A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2//merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images; and `/tmp/awf-lib` helper staging (one-shot token protection library, Claude API key helper, `gh` CLI proxy wrapper, CA bundles, runner shims) moved to `/run/awf-lib`, eliminating silent degradation under `filesystem.allowWrite`. Startup now fails closed if the one-shot token library cannot be staged or if CLI proxying is enabled but the `gh` wrapper cannot be installed. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; inspect entrypoint logs for `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` — on older AWF this confirms the silent-degradation mode; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 | -| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `buildContainerSecurityHardening` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read from `/proc/self/status` `CapBnd` or equivalent) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `cat /proc/self/status \| grep CapBnd` on the Docker daemon host / DinD sidecar and decode the bitmask, or `docker info \| grep -i cgroup` for an approximate signal; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | +| A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | ## Category B — Self-hosted runners @@ -70,7 +70,7 @@ Establish these facts before matching a failure mode: | B24 | Repeated `EACCES` retries from the harness reading `${RUNNER_TEMP}/gh-aw/mcp-config/mcp-servers.json` (or other files under `${RUNNER_TEMP}/gh-aw`) with no root-cause diagnostic explaining the native-root fallback ownership mismatch; occurs when AWF is invoked as native root (no `sudo`, so no `SUDO_UID`) | AWF falls back to a default sandbox uid/gid (`1000:1000`) when it cannot recover the original host identity from `SUDO_UID`. Files under `${RUNNER_TEMP}/gh-aw` created by the root-run harness stay root-owned, so the sandbox identity cannot read them once they are mounted read-only into the agent container | **Fixed in AWF (PR github/gh-aw-firewall#7565, merged 2026-08-20):** `isNativeRootWithoutSudo()` (`src/host-identity.ts`) detects root execution without `SUDO_UID` and logs an explicit warning identifying the fallback sandbox identity. `repairRunnerTempGhAwOwnership()` (`src/config-writer.ts`) recursively `chown`s `${RUNNER_TEMP}/gh-aw` to the resolved sandbox uid/gid (via `chown -h -P -R`, never following symlinks) before Docker Compose generation and container launch. Upgrade AWF to include github/gh-aw-firewall#7565. | Run AWF as root directly (no `sudo`) and check for the warning `Host process is running as root with no SUDO_UID; AWF will use sandbox identity ...`; compare `stat -c '%U:%G' "$RUNNER_TEMP/gh-aw/mcp-config/mcp-servers.json"` before/after a run — it should show the sandbox uid:gid after the fix instead of `root` | github/gh-aw-firewall#7564, github/gh-aw-firewall#7565 | | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | -| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | +| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | ## Category C — GHES / GHEC / `ghe.com` diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 5156e6e46..fa776eeaa 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -152,7 +152,7 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('| `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26'); // B27 new failure mode (stale awf-net collision on persistent self-hosted runners) expect(content).toContain('| B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project'); - expect(content).toContain('`generateDockerCompose()` in `src/docker-manager.ts` names the Docker network `awf-net`'); + expect(content).toContain('`generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net`'); expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):**'); expect(content).toContain('github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817'); expect(content).toContain('| `a network with name awf-net exists but was not created for project` | B27');