Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions .github/agents/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ Prefer the narrowest match. Examples:
- `[WARN] Rootless artifact permission repair failed ... (exit 1)` with no stderr detail, followed by `Command completed with exit code: 1` despite apparent agent success → B11 (repair container stderr not captured; chroot-home removal failure escalated to fatal exit)
- `none of the git remotes correspond to the GH_HOST environment variable` → C4
- `400 bad request: Authorization header is badly formatted` → C3
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog; also check for `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` sentinel suppressing GitHub-token auth path — fixed in github/gh-aw-firewall#6237)
- `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)
- `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware)
- `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS)

Expand All @@ -108,7 +109,9 @@ B11 / github/gh-aw-firewall#6072 — Repair container stderr not captured and ch

C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.

C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens).

D7 / github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 — Claude Code (Bun/JSC) crashes with `SIGSEGV`/`SIGABRT` under `--container-runtime gvisor` because JSC JIT is incompatible with gVisor's W^X memory restrictions. **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` as a manual fallback.

## Output Requirements

Expand Down Expand Up @@ -178,7 +181,7 @@ Establish these facts before matching a failure mode:
| ID | Signal | Root cause | Fix / flag | Probe | Citations |
|---|---|---|---|---|---|
| B1 | `/home/runner/...` paths are wrong on a custom runner home | The runner uses a non-standard `HOME` | Use the real `HOME`; when configuring stdin, set `chroot.identity.home` | `echo "$HOME"` and inspect mounted home paths | #2109, #2290 |
| B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy` | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975 |
| B2 | All outbound traffic fails behind a mandatory corporate proxy | AWF must chain Squid through the upstream proxy | Set `https_proxy` / `http_proxy` on the host or use `--upstream-proxy`. **Note:** AWF ≤ v0.27.32 (before PR github/gh-aw-firewall#6267) silently blocked proxy environment variables (`NO_PROXY`, `HTTP_PROXY`, `HTTPS_PROXY`, etc.) when passed via `--env` if `enableApiProxy` is active, because `PROXY_ENV_VARS` were in the credential exclusion set. **Fixed in PR github/gh-aw-firewall#6267**: proxy vars are now checked against the `PROXY_ENV_VARS` allowlist and passed through `additionalEnv` even when credential isolation is active. | `env | grep -i proxy`; inspect Squid config for `cache_peer` | #1975, github/gh-aw-firewall#6267 |
| B3 | Squid exits with `FATAL: http_port: IPv6 is not available` | Docker IPv6 is disabled but Squid tries to bind an IPv6 listener | Enable Docker/kernel IPv6 (required with current AWF builds), or use a custom AWF build that removes the `[::]` listener | `docker info | grep -i ipv6`; inspect `/proc/sys/net/ipv6/conf/all/disable_ipv6` | #2139 |
| B4 | `node: command not found` after `actions/setup-node` on self-hosted | Node was installed in `$HOME/work/_tool` and that toolcache is not visible | Mount / expose the runner toolcache; use `AWF_EXTRA_TOOLCACHE_DIRS` if needed | `which node`; inspect `$HOME/work/_tool/node` | #3544, #3545 |
| B5 | `getaddrinfo EAI_AGAIN <awmg-cli-proxy>` → `awf-cli-proxy could not connect to the external DIFC proxy` → `The agent was never invoked` in `--network-isolation` + `--topology-attach` runs | Startup ordering deadlock: `connectTopologyContainers()` runs only after `startContainers()` succeeds, but `startContainers()` blocks on the cli-proxy health gate that requires the topology peer to be reachable on `awf-net` (which `internal: true`). The peer is never attached → EAI_AGAIN → fail-fast → deadlock. Deterministic, not flaky. | Resolved in AWF: attach topology peers to `awf-net` before the health-gated bring-up (Fix A: split `up -d`, network first → attach → remaining); also harden cli-proxy to treat `EAI_AGAIN`/`ENOTFOUND` as not-yet-ready (Fix B) | Confirm `topologyAttach` is non-empty; check the cli-proxy logs for `EAI_AGAIN`; verify AWF version includes the ordering fix | #5543, #5542 |
Expand All @@ -200,7 +203,7 @@ Establish these facts before matching a failure mode:
| C5 | `malformed version:` from `gh --repo` in later user steps | `GH_HOST=localhost:18443` leaked into non-AWF steps via `$GITHUB_ENV` | Primary fix belongs in gh-aw; cli-proxy can mitigate only partially | Check `$GITHUB_ENV` and `curl http://localhost:18443/api/v3/meta` | #3937 |
| C6 | Safe-outputs post-processing talks to github.com instead of GHES | gh-aw emitted `GH_HOST` to the wrong channel for later jobs | Fix the compiler / environment propagation in gh-aw | Inspect `$GITHUB_OUTPUT` and `$GITHUB_ENV` for `GH_HOST` | #1460, #1566 |
| C7 | `awf-cli-proxy` DIFC-proxy liveness probe loops retrying; cli-proxy logs show `diagnosis=unknown` (AWF < v0.27.12) or `diagnosis=reachable-but-api-error (HTTP NNN)` with a `*.ghe.com` hint (AWF ≥ v0.27.12); AWF fails to start | DIFC proxy is reachable but the forwarded `gh api rate_limit` call returns an HTTP error because the DIFC proxy is not enterprise-host-aware on data-residency `*.ghe.com` tenants | **Partially mitigated**: upgrade to AWF ≥ v0.27.12 for a targeted `*.ghe.com` hint and HTTP status in cli-proxy logs; root cause (DIFC proxy enterprise-host awareness) is **unresolved** in companion projects (github/gh-aw-mcpg#8202, github/gh-aw#41911) | Check `GITHUB_SERVER_URL` for `*.ghe.com`; inspect cli-proxy logs for `diagnosis=unknown` or `reachable-but-api-error (HTTP NNN)`; confirm AWF ≥ v0.27.12 for the targeted hint | #5615, #5616 |
| C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16. | `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including github/gh-aw-firewall#5872. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872 |
| C8 | `400 bad request: Authorization header is badly formatted` on **GHEC (`*.ghe.com`)** runners when `COPILOT_API_TARGET=api.business.githubcopilot.com`; Copilot Business calls receive `Bearer` instead of required `token` prefix. Reproduced on AWF v0.27.13 and v0.27.16; or `400` persists even after upgrading past github/gh-aw-firewall#5872 when `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is set by `gh-aw` offline mode | Two distinct root causes: **(1) Pre-#5872:** `copilotTargetRequiresGitHubTokenPrefix()` checked `AWF_PLATFORM_TYPE` guard first. On GHEC, AWF auto-injects `AWF_PLATFORM_TYPE=ghec`, which short-circuited to `false` before querying the `GITHUB_TOKEN_PREFIX_COPILOT_TARGETS` catalog. **(2) Post-#5872 / #6237:** `gh-aw` offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel; AWF before #6237 treated it as a real BYOK key, which took precedence over and suppressed the GitHub-token auth path | **Fixed in AWF (PR github/gh-aw-firewall#5872)**: catalog endpoints (`api.enterprise.githubcopilot.com`, `api.business.githubcopilot.com`) are now checked first (always `token`); the platform-type guard now only affects the GHES heuristic for unknown targets. Upgrade to AWF version including github/gh-aw-firewall#5872. **Additional fix (PR github/gh-aw-firewall#6237):** treats `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens), restoring the GitHub-token auth path on Business/Enterprise targets. | `awf --version`; inspect api-proxy logs for 400 on `api.business.githubcopilot.com`; confirm `AWF_PLATFORM_TYPE=ghec` is set; check whether `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is present | github/gh-aw-firewall#5871, github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 |

## Category D — Alternative runtimes and adjacent gaps

Expand All @@ -212,6 +215,7 @@ Establish these facts before matching a failure mode:
| D4 | Enterprise LLM gateway needs an injected auth header | API proxy lacks a user extension point for that hop | Known unresolved proposal | No general probe; capture the required header flow in the report | #4849 |
| D5 | gVisor install step exits with `HTTP 404` / download failure; `runsc` binary not found; AWF exits with `runtime 'gvisor' is not available` or compose up fails immediately after `--container-runtime gvisor` | The `gh-aw`-compiler-generated gVisor install step (or smoke-test lock files) pins a specific gVisor release tag (e.g. `20250623.0`); if that artifact is no longer available from `storage.googleapis.com`, the download returns 404 | Manually update the affected generated `.lock.yml` install step to a currently available release (for example, `20250707.0`), or upgrade to a `gh-aw` release containing the corresponding `DefaultGVisorVersion` update and recompile. PR github/gh-aw-firewall#6143 only patched this repository's four smoke-test lock files; it did not add an AWF configuration option or release an AWF fix. | `ARCH=$(uname -m); curl -sI https://storage.googleapis.com/gvisor/releases/release/20250623.0/${ARCH}/runsc` — HTTP 404 confirms the pinned artifact is unavailable; replace the tag with an available release and retry | github/gh-aw-firewall#6143 |
| D6 | AWF exits with `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` when `--container-runtime sbx` is set; agent never starts; infrastructure containers (Squid, api-proxy) start normally but no microVM is created | `--container-runtime sbx` added in AWF PR github/gh-aw-firewall#6101 (merged 2026-07-11) requires the `sbx` CLI on the runner; `isSbxAvailable()` preflight fails if `sbx` is absent | Install the `sbx` CLI on the runner; see github/gh-aw-firewall#6117 for productionization and CI runner coverage. AWF injects `HTTP_PROXY`/`HTTPS_PROXY` to route HTTP(S) egress through Squid; standard domain whitelisting applies. | `command -v sbx && sbx version` — absence confirms; `docker info` | github/gh-aw-firewall#6101, github/gh-aw-firewall#6117 |
| D7 | Claude Code (Bun runtime) crashes with `SIGSEGV` / `SIGABRT` under `--container-runtime gvisor`; the harness retries multiple times, each attempt failing with exit code 1; no output or minimal output is produced | gVisor restricts the W^X (write XOR execute) memory operations required by JIT compilers. Bun uses JavaScriptCore (JSC) with JIT enabled by default; JSC generates native code via JIT, triggering SIGSEGV/SIGABRT on gVisor's restricted syscall surface | **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` (or set it as a job-level env var with `--env-all`) as a manual fallback. | Inside the AWF gVisor agent container: `echo $BUN_JSC_useJIT` should be `0`; a crash without this flag confirms the mode via `SIGSEGV`/`SIGABRT` signal in agent logs | github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 |

## Error-string quick lookup

Expand All @@ -229,7 +233,7 @@ Establish these facts before matching a failure mode:
| `node: command not found` on self-hosted or DinD | A8 or B4 |
| `none of the git remotes correspond to the GH_HOST environment variable` | C4 |
| `malformed version:` from `gh --repo` | C5 |
| `400 bad request: Authorization header is badly formatted` | C3 (general GHES header assembly, AWF ≤ v0.27.1); also C8 if on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` |
| `400 bad request: Authorization header is badly formatted` | C3 (general GHES header assembly, AWF ≤ v0.27.1); also C8 if on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` or if `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` is present (fixed in github/gh-aw-firewall#6237) |
| `EACCES: permission denied, mkdir` on a `/tmp/gh-aw/...` path before containers start (pre-flight) | B8 |
| `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 |
| `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` | A17 |
Expand All @@ -244,6 +248,7 @@ Establish these facts before matching a failure mode:
| `diagnosis=unknown` from `awf-cli-proxy` DIFC probe (proxy reachable, no connection error) with `GITHUB_SERVER_URL=*.ghe.com`, or `diagnosis=reachable-but-api-error (HTTP NNN)` | C7 |
| `HTTP 404` / `404 Not Found` downloading `runsc` from `storage.googleapis.com` during gVisor install | D5 |
| `Docker sbx CLI not found. Install sbx to use --container-runtime sbx.`, `Error: sbx is not available`, or `spawn sbx ENOENT` with `--container-runtime sbx` | D6 |
| `SIGSEGV` / `SIGABRT` signal crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail | D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) |

## Known unresolved items

Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/self-hosted-runner-doctor.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,9 +112,10 @@ Prefer the narrowest match. Examples:
- `[WARN] Rootless artifact permission repair failed ... (exit 1)` with little/no stderr detail, plus cleanup warnings around chroot-home removal and `Command completed with exit code: 1` → B11 (repair warning lacked stderr context; non-zero exit originates from agent command, not cleanup)
- `none of the git remotes correspond to the GH_HOST environment variable` → C4
- `400 bad request: Authorization header is badly formatted` → C3
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog)
- `400 bad request: Authorization header is badly formatted` on `*.ghe.com` with `COPILOT_API_TARGET=api.business.githubcopilot.com` → C8 (platform-type guard short-circuits token-prefix catalog; also check for `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` sentinel suppressing GitHub-token auth path — fixed in github/gh-aw-firewall#6237)
- `diagnosis=unknown` (proxy reachable, no connection error) or `reachable-but-api-error` from DIFC probe with `GITHUB_SERVER_URL=*.ghe.com` → C7 (DIFC proxy not enterprise-host-aware)
- `Error: invalid key 'build-tools'` with `--image-tag build-tools=sha256:...` → A17 (build-tools not in IMAGE_DIGEST_KEYS)
- `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)

### 4. Check for known gaps and notable fixes

Expand All @@ -136,7 +137,9 @@ B11 / github/gh-aw-firewall#6072 — Rootless permission-repair diagnostics were

C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved.

C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872.
C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens).

D7 / github/gh-aw-firewall#6260, github/gh-aw-firewall#6261, github/gh-aw-firewall#6276 — Claude Code (Bun/JSC) crashes with `SIGSEGV`/`SIGABRT` under `--container-runtime gvisor` because JSC JIT is incompatible with gVisor's W^X memory restrictions. **AWF (PR github/gh-aw-firewall#6276) automatically sets `BUN_JSC_useJIT=0`** at runtime via `buildToolEnvironment()` when Claude runs under gVisor — no workflow change required. For older AWF builds without github/gh-aw-firewall#6276, pass `--env BUN_JSC_useJIT=0` as a manual fallback.

### 5. Avoid duplicate triage

Expand Down
Loading
Loading