docs: add B31 runner-doctor guidance for sudo secure_path PATH shadowing - #8218
Merged
Conversation
Copilot
AI
changed the title
[WIP] Update Runner Doctor knowledge-base for docker-sudo-iptables lesson
docs: add B31 runner-doctor guidance for sudo secure_path PATH shadowing
Sep 6, 2026
lpcox
marked this pull request as ready for review
September 6, 2026 16:05
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The generated workflow lock is stale, B31 lacks alignment-test coverage, and its PR resolution status is outdated.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds B31 guidance for diagnosing sudo secure_path toolchain shadowing.
Changes:
- Documents B31’s signal, cause, mitigation, and probe.
- Mirrors guidance across workflow and portable-agent documentation.
- Adds quick-lookup entries.
File summaries
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds canonical B31 guidance. |
.github/workflows/self-hosted-runner-doctor.md |
Adds workflow lookup guidance. |
.github/agents/self-hosted-runner-doctor.md |
Mirrors B31 in the portable agent. |
Review details
Suppressed comments (2)
.github/agents/self-hosted-runner-doctor.md:317
- This copied B31 row also records #8173 as an unmerged draft and says no production fix was needed, but #8173 merged on 2026-09-05 and
github/gh-aw#58458was closed by merged production fixgithub/gh-aw#58625. Synchronize this row with the corrected current resolution.
| B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`; sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin dir precedence | **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars like `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior, since it was previously only exercised via mocks. | Compare `sudo -E awf --env-all -- env` output for `PATH` before/after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir (from `$GITHUB_PATH`) appears ahead of `/usr/bin` in `AWF_HOST_PATH` — if not, this is a genuine regression, not this known-working mode | github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |
.github/workflows/shared/self-hosted-failure-modes.md:80
- Add B31 assertions to
scripts/ci/self-hosted-runner-doctor-workflow.test.ts. That alignment test explicitly guards the B29/B30 catalog rows and quick lookups (lines 179–183 and 202–204), but currently does not assert this new mode, so future changes can silently desynchronize the shared catalog, workflow playbook, and portable agent.
| B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`; sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin dir precedence | **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars like `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior, since it was previously only exercised via mocks. | Compare `sudo -E awf --env-all -- env` output for `PATH` before/after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir (from `$GITHUB_PATH`) appears ahead of `/usr/bin` in `AWF_HOST_PATH` — if not, this is a genuine regression, not this known-working mode | github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 |
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) | ||
| - `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021) | ||
| - `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30 (check preserved logs dir for `awf-startup-error.json`; fixed in github/gh-aw-firewall#8023) | ||
| - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` strips `$GITHUB_PATH`-augmented `PATH` before AWF sees it; already mitigated on `main` via `recoverHostPaths()` reading `$GITHUB_PATH` directly — not a new bug, but a diagnostic to rule out before suspecting AWF) |
|
|
||
| B30 / github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 — A pre-egress AWF startup failure can leave no Squid `access.log`, causing `awf logs summary`/`stats` to report only "no log sources found". **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes redacted `awf-startup-error.json`; log discovery and stats/summary recognize it. Check the preserved proxy-logs directory for that file and upgrade AWF to include github/gh-aw-firewall#8023. | ||
|
|
||
| B31 / github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 — Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (for example `ruby/setup-ruby` choosing Ruby 3.4.8) can be shadowed by the system-installed version (for example `/usr/bin/ruby` 3.2.3) inside the AWF agent container even when `--env-all`/`AWF_HOST_PATH` capture is active. `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`, and sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin-dir precedence. **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars such as `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior because it was previously only exercised via mocks. Compare `sudo -E awf --env-all -- env` output for `PATH` before and after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir from `$GITHUB_PATH` appears ahead of `/usr/bin` in `AWF_HOST_PATH`. If it does not, this is a genuine regression rather than this known-working mode. |
| | B29 | `codex`-engine (and similar) workflows abort with `report_incomplete`: "context-rebuild circuit breaker tripped" after repeatedly failing to `cd` into the expected workspace path (for example `/home/runner/work/<repo>/<repo>: No such file or directory`) | `--container-workdir` sets the agent's starting directory to a host-style absolute path, but that path was not guaranteed to be bind-mounted inside the chroot. If it was outside the workspace mount, `/tmp`, system mounts, `$HOME` tool mounts, or an explicit `--mount`, `entrypoint.sh` silently fell back to `/`, causing repeated context-rebuild retries | **Fixed in AWF (PR github/gh-aw-firewall#8021, merged 2026-09-02):** `buildContainerWorkDirMounts()` in `src/services/agent-volumes/workspace-mounts.ts` emits an explicit `<workdir>:<workdir>:rw` bind mount when the configured workdir is not already reachable inside the chroot. It refuses paths inside deliberately hidden roots and warns when the host directory does not exist. Upgrade AWF to include github/gh-aw-firewall#8021. | Inspect generated `docker-compose.yml` for a bind mount matching `--container-workdir` when no other mount covers it; check startup logs for a workdir-not-found warning instead of a silent `/` fallback; reproduce with a workdir outside every default mount | github/gh-aw-firewall#8015, github/gh-aw-firewall#8021 | | ||
| | B30 | AWF-sandbox workflows fail before Squid starts (for example from a bad bind-mount spec), leaving no Squid `access.log`; `awf logs summary`/`awf logs stats` report only "no log sources found" | AWF had no mechanism to preserve startup-phase failure detail when containers never produced Squid logs, so the underlying cause was lost | **Fixed in AWF (PR github/gh-aw-firewall#8023, merged 2026-09-02):** AWF writes a redacted `awf-startup-error.json` (timestamp, phase, failure message) into the proxy logs directory on startup abort; log discovery recognizes it via `AWF_LOGS_DIR` and preserved `/tmp/squid-logs-*` discovery, and stats/summary include the diagnostic. Upgrade AWF to include github/gh-aw-firewall#8023. | After a pre-egress failure, check the preserved proxy-logs directory for `awf-startup-error.json`; run `awf logs summary` and confirm it surfaces the startup diagnostic | github/gh-aw-firewall#8014, github/gh-aw-firewall#8023 | | ||
|
|
||
| | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invokes AWF via `sudo -E awf ...`; sudoers' `secure_path` can silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF ever observes `process.env.PATH`, so a naive PATH capture loses the hosted-toolcache bin dir precedence | **Already handled on `main`, not a bug:** `readGitHubPathEntries()` reads `$GITHUB_PATH` directly from disk (unaffected by `secure_path`), and `recoverHostPaths()` merges those entries into `AWF_HOST_PATH` ahead of the (possibly stripped) `PATH`, also recovering toolchain vars like `GOROOT`/`JAVA_HOME` from `$GITHUB_ENV` when running as sudo. No production fix was needed; PR github/gh-aw-firewall#8173 (draft, not yet merged) adds a regression test (`src/services/agent-environment/host-path-recovery.test.ts`) covering this composed sudo-boundary behavior, since it was previously only exercised via mocks. | Compare `sudo -E awf --env-all -- env` output for `PATH` before/after crossing the sudo boundary under `docker-sudo-iptables`; confirm the hosted-toolcache bin dir (from `$GITHUB_PATH`) appears ahead of `/usr/bin` in `AWF_HOST_PATH` — if not, this is a genuine regression, not this known-working mode | github/gh-aw#58458, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 | |
Collaborator
|
@copilot address review feedback |
Contributor
Author
Contributor
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Contributor
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The self-hosted runner doctor catalog did not include the
docker-sudo-iptables+sudoers secure_pathcase where a setup action’s toolchain wins are silently shadowed by the system Ruby/Python/etc. in the AWF agent. This was a known, already-mitigated path-loss issue rather than a new production bug, but it needed a clear diagnosis entry to prevent false alarms.Summary
sandbox.agent.runtime: docker-sudo-iptables.sudo/secure_pathstripping the runner’s$GITHUB_PATH-augmentedPATHbefore AWF records it, and that the fix is already handled onmainvia direct$GITHUB_PATHrecovery.Changes
.github/workflows/shared/self-hosted-failure-modes.mdwith the new B31 entry, including signal, root cause, mitigation, probe, and citations..github/workflows/self-hosted-runner-doctor.md..github/agents/self-hosted-runner-doctor.mdso the portable agent stays aligned with the workflow playbook and shared catalog.Why this matters
secure_path-stripping case and distinguish it from a true AWF regression.mainmitigation story: host toolchain entries are recovered from$GITHUB_PATH/$GITHUB_ENVbefore the strippedPATHis used, so the diagnosis stays precise and actionable.