Document new Squid DNS and log-permission failure modes - #8290
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The documentation records an incorrect DNS timeout and inaccurately describes the previous Squid log repair.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds runner-doctor guidance for Squid DNS caching and log-permission failures.
Changes:
- Documents B32 and B33 failure modes.
- Mirrors diagnostic guidance in workflow and portable playbooks.
- Extends alignment tests for both entries.
File summaries
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds canonical B32/B33 guidance. |
.github/workflows/self-hosted-runner-doctor.md |
Adds quick-lookup entries. |
.github/agents/self-hosted-runner-doctor.md |
Mirrors catalog and lookup updates. |
scripts/ci/self-hosted-runner-doctor-workflow.test.ts |
Adds alignment assertions. |
Review details
Suppressed comments (2)
.github/workflows/shared/self-hosted-failure-modes.md:138
- The quick lookup repeats the inaccurate claim that an ownership repair existed before #8251. The previous shutdown helper only changed mode bits; #8251 added the
chownthat transfers ownership to the runner.
| `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact | B33 (Squid log ownership repair only ran at shutdown; fixed in github/gh-aw-firewall#8251 with reusable pre-diagnostic `chown`+`chmod` repair) |
.github/agents/self-hosted-runner-doctor.md:321
- The old repair was not an ownership repair: it only ran
chmod, and #8251 added the runner-UIDchown. The current wording hides the reason artifact preservation still encounteredEPERM.
| B33 | `[DEBUG] Could not check Squid logs: EACCES ... access.log` during diagnostics, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, although logs are intact | Squid writes logs as UID 13; ownership repair previously ran only at shutdown, after mid-run diagnostics and before the preservation `chmod` | **Fixed in AWF (PR github/gh-aw-firewall#8251, merged 2026-09-07):** reusable `fixSquidLogPermissions()` now `chown`s to the runner UID/GID via `docker exec -e` and `chmod`s; `runAgentCommand()` repairs permissions before `checkSquidLogs()`, and preserved-log `chmod` is reported separately after rename. Upgrade AWF to include github/gh-aw-firewall#8251. | Trigger a blocked-domain or upstream-error diagnostic and confirm `access.log` is readable; after the run, `ls -la <preserved-squid-logs-dir>` should show runner-UID ownership and no `Could not check Squid logs`/`Could not preserve squid logs` messages | github/gh-aw-firewall#8249, github/gh-aw-firewall#8251 |
- Files reviewed: 4/4 changed files
- Comments generated: 8
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 | | ||
|
|
||
| | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invoked AWF via `sudo -E awf ...`; sudoers' `secure_path` could silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF observed `process.env.PATH`, losing hosted-toolcache bin-dir precedence | **Fixed in gh-aw (PR github/gh-aw#58625, merged 2026-09-05):** privileged AWF startup preserves the caller `PATH`. AWF's `readGitHubPathEntries()`/`recoverHostPaths()` recovery remains defense in depth, and merged PR github/gh-aw-firewall#8173 adds regression coverage in `src/services/agent-environment/host-path-recovery.test.ts`; no AWF production-code change was needed. | Confirm the setup-action hosted-toolcache bin dir remains ahead of `/usr/bin` in `AWF_HOST_PATH` under `docker-sudo-iptables`; if it does not, this is a regression. | github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 | | ||
| | B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted `api.github.com`/`github.com` traffic with `403` or DNS `SERVFAIL`, recurring across otherwise-healthy runs | Squid's default `negative_dns_ttl` is 1 minute, so one transient upstream `SERVFAIL` is negatively cached and replayed for up to 60 seconds even after DNS recovers | **Fixed in AWF (PR github/gh-aw-firewall#8171, merged 2026-09-05):** `generateDnsSection()` emits `negative_dns_ttl 1 seconds` and `dns_timeout 5 seconds`. Upgrade AWF to include github/gh-aw-firewall#8171. | Inspect generated `squid.conf` for `negative_dns_ttl 1 seconds`, `dns_timeout 5 seconds`, and `dns_nameservers`; correlate Squid `TCP_DENIED`/SERVFAIL bursts with concurrent startup or resolver load on unpatched AWF | github/gh-aw-firewall#8168, github/gh-aw-firewall#8171 | |
There was a problem hiding this comment.
Addressed in commit 53717a1: aligned the shared and portable runner-doctor guidance with negative_dns_ttl 1 seconds, dns_retransmit_interval 1 seconds, and dns_timeout 10 seconds, corrected the pre-#8251 repair to chmod-only, and strengthened the alignment assertions. The targeted test passes.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@copilot address this feedback #8290 (comment) |
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (2 files)
Coverage comparison generated by |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.
|
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅
|
|
🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅ Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
|
📰 DEVELOPING STORY: Smoke Docker Sbx reports failed. Our correspondents are investigating the incident...
|
|
❌ Smoke Claude failed Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "api.anthropic.com"See Network Configuration for more information.
|
|
✅ Build Test Suite completed successfully!
|
|
🚀 Security Guard has started processing this pull request |
Smoke Test: Copilot BYOK (Direct) Mode✅ All tests passed
Status: PASS — Direct BYOK mode (COPILOT_PROVIDER_API_KEY) operational
|
|
EGRESS_RESULT allow=pass deny=pass ✅ Allowed domain (github.com) reachable: allowed=200 Overall: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "example.com"See Network Configuration for more information.
|
Smoke Test: Services Connectivity
Overall: PASS
|
github-actions
Bot
added
smoke-copilot-network-isolation
|
Smoke Test: Copilot Engine — ✅ PASS
Recent merged PRs checked:
Overall: PASS cc
|
|
|
|
Recent PRs: fix: send delegation TTLs in seconds; chore(deps): Bump the all-github-actions group with 21 updates — ❌
|
Chroot Version Comparison Results
Overall: FAILED — Node.js version mismatch between host and chroot environment.
|
Smoke Test: API Proxy OTEL Tracing — Results
Overall: ✅ All 5 scenarios pass. OTEL module init, span creation, GenAI usage attributes, env var/context propagation, and OTLP export all validated successfully for PR #8290.
|
|
Smoke test:
Warning Firewall blocked 7 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "msfeed25.pkgs.visualstudio.com"
- "www.google.com"
- "www.gstatic.com"See Network Configuration for more information.
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — PASS Notes: Maven's default
|
Adds runner-doctor guidance for two recurring self-hosted runner failures:
SERVFAILresponses were replayed by Squid’s default 60-second negative DNS cache.EACCES/EPERMerrors.Changes
Example diagnostic signals now covered: