Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/agents/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ Prefer the narrowest match. Examples:
- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)
- `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021)
- `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30 (check preserved logs dir for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023)
- A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` strips `$GITHUB_PATH`-augmented `PATH` before AWF sees it; already mitigated on `main` via `recoverHostPaths()` reading `$GITHUB_PATH` directly — not a new bug, but a diagnostic to rule out before suspecting AWF)
- Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038)

### 4. Check for known gaps and notable fixes
Expand Down Expand Up @@ -186,6 +187,8 @@ B29 / github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 — `codex`-engine

B30 / github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 — A pre-egress AWF startup failure can leave no Squid `access.log`, causing `awf logs summary`/`stats` to report only "no log sources found". **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes redacted `awf-startup-error.json`; log discovery and stats/summary recognize it. Check the preserved proxy-logs directory for that file and upgrade AWF to include github/gh-aw-firewall#8023.

B31 / github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 — Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (for example `ruby/setup-ruby` choosing Ruby 3.4.8) can be shadowed by the system-installed version (for example `/usr/bin/ruby` 3.2.3) inside the AWF agent container even when `--env-all`/`AWF_HOST_PATH` capture is active. `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`, and sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin-dir precedence. **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars such as `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior because it was previously only exercised via mocks. Compare `sudo -E awf --env-all -- env` output for `PATH` before and after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir from `$GITHUB_PATH` appears ahead of `/usr/bin` in `AWF_HOST_PATH`. If it does not, this is a genuine regression rather than this known-working mode.

D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor.

D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime.
Expand Down Expand Up @@ -311,6 +314,7 @@ Establish these facts before matching a failure mode:

| B29 | `codex`-engine (and similar) workflows abort with `report_incomplete`: "context-rebuild circuit breaker tripped" after repeatedly failing to `cd` into the expected workspace path (for example `/home/runner/work/<repo>/<repo>: No such file or directory`) | `--container-workdir` sets the agent's starting directory to a host-style absolute path, but that path was not guaranteed to be bind-mounted inside the chroot. If it was outside the workspace mount, `/tmp`, system mounts, `$HOME` tool mounts, or an explicit `--mount`, `entrypoint.sh` silently fell back to `/`, causing repeated context-rebuild retries | **Fixed in AWF (PR github/gh-aw-firewall#8021, merged 2026-09-02):** `buildContainerWorkDirMounts()` in `src/services/agent-volumes/workspace-mounts.ts` emits an explicit `<workdir>:<workdir>:rw` bind mount when the configured workdir is not already reachable inside the chroot. It refuses paths inside deliberately hidden roots and warns when the host directory does not exist. Upgrade AWF to include github/gh-aw-firewall#8021. | Inspect generated `docker-compose.yml` for a bind mount matching `--container-workdir` when no other mount covers it; check startup logs for a workdir-not-found warning instead of a silent `/` fallback; reproduce with a workdir outside every default mount | github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 |
| B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 |
| B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`; sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin dir precedence | **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars like `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior, since it was previously only exercised via mocks. | Compare `sudo -E awf --env-all -- env` output for `PATH` before/after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir (from `$GITHUB_PATH`) appears ahead of `/usr/bin` in `AWF_HOST_PATH` — if not, this is a genuine regression, not this known-working mode | github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |

## Category C — GHES / GHEC / `ghe.com`

Expand Down
1 change: 1 addition & 0 deletions .github/workflows/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ Prefer the narrowest match. Examples:
- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)
- `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021)
- `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30 (check preserved logs dir for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023)
- A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` strips `$GITHUB_PATH`-augmented `PATH` before AWF sees it; already mitigated on `main` via `recoverHostPaths()` reading `$GITHUB_PATH` directly — not a new bug, but a diagnostic to rule out before suspecting AWF)
- Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038)
- `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)

Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/shared/self-hosted-failure-modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ Establish these facts before matching a failure mode:
| B29 | `codex`-engine (and similar) workflows abort with `report_incomplete`: "context-rebuild circuit breaker tripped" after repeatedly failing to `cd` into the expected workspace path (for example `/home/runner/work/<repo>/<repo>: No such file or directory`) | `--container-workdir` sets the agent's starting directory to a host-style absolute path, but that path was not guaranteed to be bind-mounted inside the chroot. If it was outside the workspace mount, `/tmp`, system mounts, `$HOME` tool mounts, or an explicit `--mount`, `entrypoint.sh` silently fell back to `/`, causing repeated context-rebuild retries | **Fixed in AWF (PR github/gh-aw-firewall#8021, merged 2026-09-02):** `buildContainerWorkDirMounts()` in `src/services/agent-volumes/workspace-mounts.ts` emits an explicit `<workdir>:<workdir>:rw` bind mount when the configured workdir is not already reachable inside the chroot. It refuses paths inside deliberately hidden roots and warns when the host directory does not exist. Upgrade AWF to include github/gh-aw-firewall#8021. | Inspect generated `docker-compose.yml` for a bind mount matching `--container-workdir` when no other mount covers it; check startup logs for a workdir-not-found warning instead of a silent `/` fallback; reproduce with a workdir outside every default mount | github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 |
| B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 |

| B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`; sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin dir precedence | **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars like `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior, since it was previously only exercised via mocks. | Compare `sudo -E awf --env-all -- env` output for `PATH` before/after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir (from `$GITHUB_PATH`) appears ahead of `/usr/bin` in `AWF_HOST_PATH` — if not, this is a genuine regression, not this known-working mode | github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |

## Category C — GHES / GHEC / `ghe.com`

| ID | Signal | Root cause | Fix / flag | Probe | Citations |
Expand Down Expand Up @@ -129,6 +131,7 @@ Establish these facts before matching a failure mode:
| `400 bad request: Authorization header is badly formatted` on derived `copilot-api.*.ghe.com` target specifically (not `api.business.githubcopilot.com`) | C9 (derived GHEC Copilot API target missing GitHub `token` prefix; fixed in github/gh-aw-firewall#6991) |
| `context-rebuild circuit breaker tripped` together with repeated failed `cd` into the expected workspace path | B29 (container-workdir was not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021) |
| `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` | B30 (check the preserved logs directory for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023) |
| A setup-action-selected toolchain version (for example `ruby/setup-ruby` picking Ruby 3.4.8) is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` | B31 (sudoers `secure_path` strips `$GITHUB_PATH`-augmented `PATH` before AWF sees it; already mitigated on `main` via `recoverHostPaths()` reading `$GITHUB_PATH` directly — not a new bug, but a diagnostic to rule out before suspecting AWF) |
| Copilot on Business/Enterprise/GHEC uses the wrong Authorization scheme specifically with a fine-grained PAT (`github_pat_...`) | C10 (credential-kind scheme selection, GHEC model discovery, and legacy placeholder handling fixed in github/gh-aw-firewall#8038) |
| `EACCES: permission denied, mkdir` on a `/tmp/gh-aw/...` path before containers start (pre-flight) | B8 |
| `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 |
Expand Down