docs: add B28 runner-doctor coverage for private-CA api-proxy TLS failures

.github/agents/self-hosted-runner-doctor.md

@@ -116,6 +116,7 @@ Prefer the narrowest match. Examples:
 - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679)
 - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795)
 - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817)
+- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)
 
 ### 4. Check for known gaps and notable fixes
 
@@ -294,6 +295,7 @@ Establish these facts before matching a failure mode:
 | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R <uid>:<gid> <workdir>` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 |
 | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download <id>` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 |
 | B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-<id>"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 |
+| B28 | Custom `apiProxy` targets pointing at an internal/corporate LLM router (`--openai-api-target`, `--anthropic-api-target`, etc.) fail TLS verification when the upstream endpoint's certificate chains to a private or corporate CA not present in the api-proxy sidecar's trust store | The `containers/api-proxy` Node.js sidecar had no supported way to extend its trust store for custom upstream targets; the only workarounds were disabling certificate verification (insecure) or baking a custom CA into a rebuilt image | **Fixed in AWF (PR github/gh-aw-firewall#7816, merged 2026-08-28):** new `apiProxy.caCert` config field and `--api-proxy-ca-cert <path>` CLI flag bind-mount the host CA file read-only into the api-proxy container at `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and set `NODE_EXTRA_CA_CERTS` to that path, so Node trusts the additional CA alongside its built-in roots without disabling verification. Upgrade AWF to include github/gh-aw-firewall#7816. | Inspect generated `docker-compose.yml` for the api-proxy service — a read-only bind mount to `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and `NODE_EXTRA_CA_CERTS` env var confirm the fix is active; reproduce the failure pre-fix with `awf --openai-api-target <internal-host> --allow-domains <internal-host> -- <cmd>` against an endpoint using a private-CA certificate | github/gh-aw-firewall#7807, github/gh-aw-firewall#7816 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -384,6 +386,7 @@ Establish these facts before matching a failure mode:
 | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure Blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) |
 | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) |
 | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) |
+| TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) |
 | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |
 | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 |
 | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 |

.github/workflows/self-hosted-runner-doctor.md

@@ -144,6 +144,7 @@ Prefer the narrowest match. Examples:
 - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679)
 - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795)
 - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817)
+- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)
 - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)
 
 ### 4. Check for known gaps and notable fixes

.github/workflows/shared/self-hosted-failure-modes.md

@@ -71,6 +71,7 @@ Establish these facts before matching a failure mode:
 | B25 | On native-root runners (root, no `SUDO_UID` — e.g. AWS CodeBuild), the job silently exits 0 with no output even after PR github/gh-aw-firewall#7565's `${RUNNER_TEMP}/gh-aw` ownership repair; the agent never writes any file to the checkout | PR github/gh-aw-firewall#7565 repaired `${RUNNER_TEMP}/gh-aw` ownership for native-root runners but left `config.containerWorkDir` (the checkout, `--container-workdir "$GITHUB_WORKSPACE"`) root-owned while the agent runs as the fallback sandbox identity (uid 1000). The workspace is writable by *mount* but not by *ownership*, so every agent write silently fails and the job exits 0 — a false green costing a full agent session per run | **Fixed in AWF (PR github/gh-aw-firewall#7599, merged 2026-08-21):** `src/config-writer.ts` adds `repairContainerWorkDirOwnership(config)`, called from `writeConfigs()` alongside `repairRunnerTempGhAwOwnership()`; applies to `config.containerWorkDir` only when `isNativeRootWithoutSudo()` is true. Also adds `isDirectoryWritableByIdentity()` as a post-repair preflight (checks owner/group/other mode bits directly, since `access(2)` always succeeds as root) — throws with an explicit `chown -R <uid>:<gid> <workdir>` suggestion instead of silently proceeding. Fixing ownership also resolves git's `dubious ownership` error without needing `safe.directory`. Upgrade AWF to include github/gh-aw-firewall#7599. | Run AWF as native root (no `sudo`) on a runner where the checkout is root-owned; on unpatched AWF the job exits 0 with no agent writes; on patched AWF, check for either successful chown-and-proceed, or the explicit failure message `Host workspace is not writable by the sandbox identity (<uid>:<gid>): <path>` | github/gh-aw-firewall#7593, github/gh-aw-firewall#7599 |
 | B26 | In `--network-isolation` mode, `gh api .../actions/artifacts/{id}/zip` or `gh run download` fails inside the agent sandbox with `error connecting to productionresultssa*.blob.core.windows.net`; download fails in ~350ms | Two independent, non-interacting changes: (1) `gh-aw-mcpg` PR github/gh-aw-mcpg#10350 stopped auto-following the GitHub 302 redirect for artifact ZIP requests, so the `gh` CLI (running inside cli-proxy) must follow the `Location` header itself; (2) `cli-proxy` is intentionally isolated to `awf-net` only (no `awf-ext` egress, per github/gh-aw-firewall#7066) so it has no route to `productionresultssa*.blob.core.windows.net`. Since mcpg now expects the client to follow the redirect but cli-proxy cannot reach the blob storage target directly, every ZIP download fails. | **Fixed in AWF (PR github/gh-aw-firewall#7635, merged 2026-08-22):** cli-proxy's HTTP(S) traffic is now routed through Squid; cli-proxy remains isolated from `awf-ext`; Squid ACL scopes `*.blob.core.windows.net` access to requests originating from cli-proxy's fixed IP only (`http_access allow from_cli_proxy cli_proxy_artifact_storage`), preserving blocklist precedence and SSL Bump behavior. Azure Blob storage is *not* added to the agent's general domain allowlist. Upgrade AWF to include github/gh-aw-firewall#7635. | Reproduce with `gh run download <id>` or `gh api .../actions/artifacts/{id}/zip` inside a `--network-isolation` agent sandbox; on unpatched AWF this fails within ~350ms with a blob-storage connection error; on patched AWF inspect Squid `access.log` for an ACL entry scoping `*.blob.core.windows.net` to the cli-proxy source IP | github/gh-aw#54371, github/gh-aw-firewall#7615, github/gh-aw-firewall#7635 |
 | B27 | Docker Compose refuses to start AWF containers with repeated warnings: `a network with name awf-net exists but was not created for project "awf-<id>"`, blocking container startup on a persistent (non-ephemeral) self-hosted runner | `generateDockerCompose()` in `src/compose-generator.ts` names the Docker network `awf-net` without `external: true` or a project-scoped/unique name. If a prior AWF run's network was not cleaned up (killed/timed-out process, orphaned network from a previous invocation), Compose treats the pre-existing `awf-net` as belonging to a different (or no) project and refuses to attach, since network names are not scoped per-project by default | **Fixed in AWF (PR github/gh-aw-firewall#7817, merged 2026-08-28):** AWF reclaims orphaned fixed-name Docker networks (`awf-net`) before `docker compose up`. Upgrade AWF to include github/gh-aw-firewall#7817. | `docker network ls \| grep awf-net` before a run — presence of a stale `awf-net` not associated with a live compose project reproduces the mode; on patched AWF the stale network is reclaimed automatically before compose up | github/gh-aw#56463, github/gh-aw-firewall#7809, github/gh-aw-firewall#7817 |
+| B28 | Custom `apiProxy` targets pointing at an internal/corporate LLM router (`--openai-api-target`, `--anthropic-api-target`, etc.) fail TLS verification when the upstream endpoint's certificate chains to a private or corporate CA not present in the api-proxy sidecar's trust store | The `containers/api-proxy` Node.js sidecar had no supported way to extend its trust store for custom upstream targets; the only workarounds were disabling certificate verification (insecure) or baking a custom CA into a rebuilt image | **Fixed in AWF (PR github/gh-aw-firewall#7816, merged 2026-08-28):** new `apiProxy.caCert` config field and `--api-proxy-ca-cert <path>` CLI flag bind-mount the host CA file read-only into the api-proxy container at `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and set `NODE_EXTRA_CA_CERTS` to that path, so Node trusts the additional CA alongside its built-in roots without disabling verification. Upgrade AWF to include github/gh-aw-firewall#7816. | Inspect generated `docker-compose.yml` for the api-proxy service — a read-only bind mount to `/usr/local/share/ca-certificates/awf-upstream-ca.crt` and `NODE_EXTRA_CA_CERTS` env var confirm the fix is active; reproduce the failure pre-fix with `awf --openai-api-target <internal-host> --allow-domains <internal-host> -- <cmd>` against an endpoint using a private-CA certificate | github/gh-aw-firewall#7807, github/gh-aw-firewall#7816 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -162,6 +163,7 @@ Establish these facts before matching a failure mode:
 | `error connecting to productionresultssa*.blob.core.windows.net` from `gh run download`/artifact ZIP fetch in `--network-isolation` mode | B26 (mcpg stopped auto-following the artifact redirect; cli-proxy had no route to Azure blob storage; fixed in github/gh-aw-firewall#7635 with scoped Squid ACL keyed to cli-proxy's fixed IP) |
 | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) |
 | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) |
+| TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) |
 | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |
 | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 |
 | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 |