Document new Squid DNS and log-permission failure modes

.github/agents/self-hosted-runner-doctor.md

@@ -121,6 +121,8 @@ Prefer the narrowest match. Examples:
 - `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021)
 - `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30 (check preserved logs dir for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023)
 - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` previously stripped the caller's `$GITHUB_PATH`-augmented `PATH`; fixed in gh-aw#58625, while AWF's `recoverHostPaths()` remains defense in depth)
+- Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape → B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`)
+- `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact → B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with a reusable pre-diagnostic `chown`+`chmod` repair)
 - Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038)
 
 ### 4. Check for known gaps and notable fixes
@@ -315,6 +317,8 @@ Establish these facts before matching a failure mode:
 | B29 | `codex`-engine (and similar) workflows abort with `report_incomplete`: "context-rebuild circuit breaker tripped" after repeatedly failing to `cd` into the expected workspace path (for example `/home/runner/work/<repo>/<repo>: No such file or directory`) | `--container-workdir` sets the agent's starting directory to a host-style absolute path, but that path was not guaranteed to be bind-mounted inside the chroot. If it was outside the workspace mount, `/tmp`, system mounts, `$HOME` tool mounts, or an explicit `--mount`, `entrypoint.sh` silently fell back to `/`, causing repeated context-rebuild retries | **Fixed in AWF (PR github/gh-aw-firewall#8021, merged 2026-09-02):** `buildContainerWorkDirMounts()` in `src/services/agent-volumes/workspace-mounts.ts` emits an explicit `<workdir>:<workdir>:rw` bind mount when the configured workdir is not already reachable inside the chroot. It refuses paths inside deliberately hidden roots and warns when the host directory does not exist. Upgrade AWF to include github/gh-aw-firewall#8021. | Inspect generated `docker-compose.yml` for a bind mount matching `--container-workdir` when no other mount covers it; check startup logs for a workdir-not-found warning instead of a silent `/` fallback; reproduce with a workdir outside every default mount | github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 |
 | B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 |
 | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invoked AWF via `sudo -E awf ...`; sudoers' `secure_path` could silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF observed `process.env.PATH`, losing hosted-toolcache bin-dir precedence | **Fixed in gh-aw (PR github/gh-aw#58625, merged 2026-09-05):** privileged AWF startup preserves the caller `PATH`. AWF's `readGitHubPathEntries()`/`recoverHostPaths()` recovery remains defense in depth, and merged PR github/gh-aw-firewall#8173 adds regression coverage in `src/services/agent-environment/host-path-recovery.test.ts`; no AWF production-code change was needed. | Confirm the setup-action hosted-toolcache bin dir remains ahead of `/usr/bin` in `AWF_HOST_PATH` under `docker-sudo-iptables`; if it does not, this is a regression. | github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |
+| B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted `api.github.com`/`github.com` traffic with `403` or DNS `SERVFAIL`, recurring across otherwise-healthy runs | Squid's default `negative_dns_ttl` is 1 minute, so one transient upstream `SERVFAIL` is negatively cached and replayed for up to 60 seconds even after DNS recovers | **Fixed in AWF (PR github/gh-aw-firewall#8171, merged 2026-09-05):** `generateDnsSection()` emits `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`. Upgrade AWF to include github/gh-aw-firewall#8171. | Inspect generated `squid.conf` for `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, `dns_timeout 10 seconds`, and `dns_nameservers`; correlate Squid `TCP_DENIED`/SERVFAIL bursts with concurrent startup or resolver load on unpatched AWF | github/gh-aw-firewall#8168, github/gh-aw-firewall#8171 |
+| B33 | `[DEBUG] Could not check Squid logs: EACCES ... access.log` during diagnostics, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, although logs are intact | Squid writes logs as UID 13; the previous shutdown-time repair only changed mode bits, ran after mid-run diagnostics, and never transferred ownership to the runner before artifact preservation | **Fixed in AWF (PR github/gh-aw-firewall#8251, merged 2026-09-07):** reusable `fixSquidLogPermissions()` now `chown`s to the runner UID/GID via `docker exec -e` and `chmod`s; `runAgentCommand()` repairs permissions before `checkSquidLogs()`, and preserved-log `chmod` is reported separately after rename. Upgrade AWF to include github/gh-aw-firewall#8251. | Trigger a blocked-domain or upstream-error diagnostic and confirm `access.log` is readable; after the run, `ls -la <preserved-squid-logs-dir>` should show runner-UID ownership and no `Could not check Squid logs`/`Could not preserve squid logs` messages | github/gh-aw-firewall#8249, github/gh-aw-firewall#8251 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 

.github/workflows/self-hosted-runner-doctor.md

@@ -150,6 +150,8 @@ Prefer the narrowest match. Examples:
 - `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021)
 - `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30 (check preserved logs dir for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023)
 - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` previously stripped the caller's `$GITHUB_PATH`-augmented `PATH`; fixed in gh-aw#58625, while AWF's `recoverHostPaths()` remains defense in depth)
+- Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape → B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`)
+- `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact → B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with a reusable pre-diagnostic `chown`+`chmod` repair)
 - Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038)
 - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`)
 

.github/workflows/shared/self-hosted-failure-modes.md

@@ -78,6 +78,8 @@ Establish these facts before matching a failure mode:
 | B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 |
 
 | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invoked AWF via `sudo -E awf ...`; sudoers' `secure_path` could silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF observed `process.env.PATH`, losing hosted-toolcache bin-dir precedence | **Fixed in gh-aw (PR github/gh-aw#58625, merged 2026-09-05):** privileged AWF startup preserves the caller `PATH`. AWF's `readGitHubPathEntries()`/`recoverHostPaths()` recovery remains defense in depth, and merged PR github/gh-aw-firewall#8173 adds regression coverage in `src/services/agent-environment/host-path-recovery.test.ts`; no AWF production-code change was needed. | Confirm the setup-action hosted-toolcache bin dir remains ahead of `/usr/bin` in `AWF_HOST_PATH` under `docker-sudo-iptables`; if it does not, this is a regression. | github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |
+| B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted `api.github.com`/`github.com` traffic with `403` or DNS `SERVFAIL`, recurring across otherwise-healthy runs | Squid's default `negative_dns_ttl` is 1 minute, so one transient upstream `SERVFAIL` is negatively cached and replayed for up to 60 seconds even after DNS recovers | **Fixed in AWF (PR github/gh-aw-firewall#8171, merged 2026-09-05):** `generateDnsSection()` emits `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`. Upgrade AWF to include github/gh-aw-firewall#8171. | Inspect generated `squid.conf` for `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, `dns_timeout 10 seconds`, and `dns_nameservers`; correlate Squid `TCP_DENIED`/SERVFAIL bursts with concurrent startup or resolver load on unpatched AWF | github/gh-aw-firewall#8168, github/gh-aw-firewall#8171 |
+| B33 | `[DEBUG] Could not check Squid logs: EACCES ... access.log` during diagnostics, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, although logs are intact | Squid writes logs as UID 13; the previous shutdown-time repair only changed mode bits, ran after mid-run diagnostics, and never transferred ownership to the runner before artifact preservation | **Fixed in AWF (PR github/gh-aw-firewall#8251, merged 2026-09-07):** reusable `fixSquidLogPermissions()` now `chown`s to the runner UID/GID via `docker exec -e` and `chmod`s; `runAgentCommand()` repairs permissions before `checkSquidLogs()`, and preserved-log `chmod` is reported separately after rename. Upgrade AWF to include github/gh-aw-firewall#8251. | Trigger a blocked-domain or upstream-error diagnostic and confirm `access.log` is readable; after the run, `ls -la <preserved-squid-logs-dir>` should show runner-UID ownership and no `Could not check Squid logs`/`Could not preserve squid logs` messages | github/gh-aw-firewall#8249, github/gh-aw-firewall#8251 |
 
 ## Category C — GHES / GHEC / `ghe.com`
 
@@ -132,6 +134,8 @@ Establish these facts before matching a failure mode:
 | `context-rebuild circuit breaker tripped` together with repeated failed `cd` into the expected workspace path | B29 (container-workdir was not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021) |
 | `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` | B30 (check the preserved logs directory for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023) |
 | A setup-action-selected toolchain version (for example `ruby/setup-ruby` picking Ruby 3.4.8) is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` | B31 (sudoers `secure_path` strips `$GITHUB_PATH`-augmented `PATH` before AWF sees it; already mitigated on `main` via `recoverHostPaths()` reading `$GITHUB_PATH` directly — not a new bug, but a diagnostic to rule out before suspecting AWF) |
+| Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape | B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`) |
+| `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact | B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with reusable pre-diagnostic `chown`+`chmod` repair) |
 | Copilot on Business/Enterprise/GHEC uses the wrong Authorization scheme specifically with a fine-grained PAT (`github_pat_...`) | C10 (credential-kind scheme selection, GHEC model discovery, and legacy placeholder handling fixed in github/gh-aw-firewall#8038) |
 | `EACCES: permission denied, mkdir` on a `/tmp/gh-aw/...` path before containers start (pre-flight) | B8 |
 | `No CA certificates were loaded from the system` inside AWF chroot on RHEL/Fedora/Amazon Linux | B9 |

scripts/ci/self-hosted-runner-doctor-workflow.test.ts

@@ -186,6 +186,13 @@ describe('self-hosted runner doctor workflow config', () => {
       expect(content).toContain('merged PR github/gh-aw-firewall#8173 adds regression coverage');
       expect(content).toContain('no AWF production-code change was needed.');
       expect(content).toContain('github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173');
+      expect(content).toContain('| B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted');
+      expect(content).toContain('negative_dns_ttl 1 seconds');
+      expect(content).toContain('dns_retransmit_interval 1 seconds');
+      expect(content).toContain('dns_timeout 10 seconds');
+      expect(content).toContain('github/gh-aw-firewall#8168, github/gh-aw-firewall#8171');
+      expect(content).toContain('| B33 | `[DEBUG] Could not check Squid logs: EACCES');
+      expect(content).toContain('github/gh-aw-firewall#8249, github/gh-aw-firewall#8251');
       expect(content).toContain('| C9 | `400 bad request: Authorization header is badly formatted` specifically on the **derived GHEC data-residency Copilot target**');
       expect(content).toContain('receives `token` instead of required `Bearer` prefix');
       expect(content).toContain('| C10 | Fine-grained GitHub PATs (`github_pat_...`) sent to Copilot Business, Enterprise, and canonical GHEC');
@@ -208,6 +215,12 @@ describe('self-hosted runner doctor workflow config', () => {
     expect(source).toContain('- `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29');
     expect(source).toContain('- `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30');
     expect(source).toContain('- A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31');
+    expect(source).toContain('- Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain');
+    expect(source).toContain('→ B32 (Squid\'s default 60-second `negative_dns_ttl`');
+    expect(source).toContain('`dns_retransmit_interval 1 seconds`');
+    expect(source).toContain('`dns_timeout 10 seconds`');
+    expect(source).toContain('- `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run');
+    expect(source).toContain('→ B33 (the previous shutdown-time repair only changed mode bits');
     expect(source).toContain('- Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10');
     expect(source).toContain('- `400 bad request: Authorization header is badly formatted` on derived `copilot-api.*.ghe.com` target specifically (not `api.business.githubcopilot.com`) → C9 (derived GHEC Copilot API target incorrectly using the GitHub `token` prefix instead of `Bearer`; fixed in github/gh-aw-firewall#8113)');
     expect(source).toContain('B12 / github/gh-aw-firewall#6326, github/gh-aw-firewall#6328 — On ARC/DinD, a topology-attached DIFC proxy addressed by Kubernetes Service name can remain unresolvable from DinD containers even after the ordering fix.');
@@ -260,6 +273,12 @@ describe('self-hosted runner doctor workflow config', () => {
       expect(playbook).toContain('B29 / github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 — `codex`-engine workflows can abort');
       expect(playbook).toContain('B30 / github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 — A pre-egress AWF startup failure');
       expect(playbook).toContain('C10 / github/gh-aw-firewall#8035, github/gh-aw-firewall#8038 — Fine-grained GitHub PATs');
+      expect(playbook).toContain('- Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain');
+      expect(playbook).toContain('→ B32 (Squid\'s default 60-second `negative_dns_ttl`');
+      expect(playbook).toContain('`dns_retransmit_interval 1 seconds`');
+      expect(playbook).toContain('`dns_timeout 10 seconds`');
+      expect(playbook).toContain('- `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run');
+      expect(playbook).toContain('→ B33 (the previous shutdown-time repair only changed mode bits');
     }
   });
 });