diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 804566085..672d754c4 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -116,6 +116,7 @@ Prefer the narrowest match. Examples: - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) +- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) ### 4. Check for known gaps and notable fixes @@ -294,6 +295,7 @@ Establish these facts before matching a failure mode: | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | | B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | +| B28 | Custom `apiProxy` targets pointing at an internal/corporate LLM router (`--openai-api-target`, `--anthropic-api-target`, etc.) fail TLS verification when the upstream endpoint's certificate chains to a private or corporate CA not present in the api-proxy sidecar's trust store | The `containers/api-proxy` Node.js sidecar had no supported way to extend its trust store for custom upstream targets; the only workarounds were disabling certificate verification (insecure) or baking a custom CA into a rebuilt image | **Fixed in AWF (PR github/gh-aw-firewall#7816, merged 2026-08-28):** new `apiProxy.caCert` config field and `--api-proxy-ca-cert ` CLI flag bind-mount the host CA file read-only into the api-proxy container at `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and set `NODE_EXTRA_CA_CERTS` to that path, so Node trusts the additional CA alongside its built-in roots without disabling verification. Upgrade AWF to include github/gh-aw-firewall#7816. | Inspect generated `docker-compose.yml` for the api-proxy service — a read-only bind mount to `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and `NODE_EXTRA_CA_CERTS` env var confirm the fix is active; reproduce the failure pre-fix with `awf --openai-api-target --allow-domains -- ` against an endpoint using a private-CA certificate | github/gh-aw-firewall#7807, github/gh-aw-firewall#7816 | ## Category C — GHES / GHEC / `ghe.com` @@ -384,6 +386,7 @@ Establish these facts before matching a failure mode: | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) | | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) | +| TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 2d0c85eab..09ce4a8f7 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -144,6 +144,7 @@ Prefer the narrowest match. Examples: - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) +- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) ### 4. Check for known gaps and notable fixes diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 2594ed600..6a626f464 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -71,6 +71,7 @@ Establish these facts before matching a failure mode: | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R : ` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (:): ` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 | | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download ` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 | | B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 | +| B28 | Custom `apiProxy` targets pointing at an internal/corporate LLM router (`--openai-api-target`, `--anthropic-api-target`, etc.) fail TLS verification when the upstream endpoint's certificate chains to a private or corporate CA not present in the api-proxy sidecar's trust store | The `containers/api-proxy` Node.js sidecar had no supported way to extend its trust store for custom upstream targets; the only workarounds were disabling certificate verification (insecure) or baking a custom CA into a rebuilt image | **Fixed in AWF (PR github/gh-aw-firewall#7816, merged 2026-08-28):** new `apiProxy.caCert` config field and `--api-proxy-ca-cert ` CLI flag bind-mount the host CA file read-only into the api-proxy container at `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and set `NODE_EXTRA_CA_CERTS` to that path, so Node trusts the additional CA alongside its built-in roots without disabling verification. Upgrade AWF to include github/gh-aw-firewall#7816. | Inspect generated `docker-compose.yml` for the api-proxy service — a read-only bind mount to `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and `NODE_EXTRA_CA_CERTS` env var confirm the fix is active; reproduce the failure pre-fix with `awf --openai-api-target --allow-domains -- ` against an endpoint using a private-CA certificate | github/gh-aw-firewall#7807, github/gh-aw-firewall#7816 | ## Category C — GHES / GHEC / `ghe.com` @@ -162,6 +163,7 @@ Establish these facts before matching a failure mode: | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) | | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) | +| TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 | | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 | diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index fa776eeaa..51eb7d22f 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -156,6 +156,10 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('**Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):**'); expect(content).toContain('github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817'); expect(content).toContain('| `a network with name awf-net exists but was not created for project` | B27'); + expect(content).toContain('| B28 | Custom `apiProxy` targets pointing at an internal/corporate LLM router (`--openai-api-target`, `--anthropic-api-target`, etc.) fail TLS verification when the upstream endpoint\'s certificate chains to a private or corporate CA not present in the api-proxy sidecar\'s trust store |'); + expect(content).toContain('`apiProxy.caCert` config field and `--api-proxy-ca-cert ` CLI flag'); + expect(content).toContain('github/gh-aw-firewall#7807, github/gh-aw-firewall#7816'); + expect(content).toContain('| TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28'); } expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); @@ -169,6 +173,7 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21'); expect(source).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); expect(source).toContain('- `a network with name awf-net exists but was not created for project` → B27'); + expect(source).toContain('- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)'); expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.'); expect(source).toContain('D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED`'); expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.');