Add B21/B22/B23 self-hosted failure-mode lessons to runner-doctor catalog

.github/workflows/self-hosted-runner-doctor.md

@@ -119,6 +119,9 @@ Prefer the narrowest match. Examples:
 - Azure CLI / ADO MCP auth failures with `~/.azure` missing inside AWF sandbox, or `AZURE_CONFIG_DIR`/`ADO_MCP_AUTH_TOKEN` empty inside agent despite being set on the runner → B18 (`.azure` was in `home.forbiddenSubdirs` and auth env vars were not forwarded; fixed in github/gh-aw-firewall#6690 — note: pre-AWF `az login` is not inherited; agent must perform OIDC re-login inside the sandbox, see github/gh-aw-firewall#6686)
 - `[WARN] Could not fix squid log permissions: ... Operation not permitted` appears as the last log after an unrelated primary AWF startup failure → B19 (benign rootless chmod cleanup error not demoted, obscuring the real failure; fixed in github/gh-aw-firewall#6939)
 - `ENETUNREACH ... :18443` (or default bridge gateway IP) from `awf-cli-proxy` in `--network-isolation` mode → B20 (known unresolved; #7066 dual-home approach was reverted because it granted general external egress from `cli-proxy`)
+- `unable to create native thread` / `Cannot create worker GC thread` inside the AWF agent from concurrent JVM builds → B21 (Docker `pids_limit` hardcoded to 1000 with no visibility/configurability; fixed in github/gh-aw-firewall#7150 with `--pids-limit`/`container.pidsLimit` plus `mount_host_cgroupfs()`)
+- Strict-security workflow (no `--legacy-security`) cannot reach a GitHub Actions `services:` raw-protocol port via `--enable-host-access` → B22 (strict topology has neither a `host.docker.internal` route nor a host-access bypass, and raw clients cannot use Squid; AWF needs a supported topology path and gh-aw must derive service ports)
+- `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (distinguish from the general PATH-resolution note already covered elsewhere) → B23 (gh-aw's cache-hit path skips the `/usr/local/bin/copilot` wrapper install while the compiler harness always spawns that hardcoded path; documentation-only fix in AWF, github/gh-aw-firewall#7151; durable fix tracked in github/gh-aw-firewall#7130, open)
 - `EACCES` + `unlink` on `/tmp/awf-...-chroot-home/<path>` during AWF cleanup (not upload-artifact) → B7 (rootless UID-remapped chroot-home files)
 - `EACCES: permission denied, mkdir '/tmp/gh-aw/...'` before containers start on a persistent runner → B8 (stale root-owned pre-flight dirs)
 - `FATAL: http_port: IPv6 is not available` → B3
@@ -175,6 +178,12 @@ B19 / github/gh-aw#50384, github/gh-aw-firewall#6934, github/gh-aw-firewall#6939
 
 B20 / github/gh-aw-firewall#7063, github/gh-aw-firewall#7066 — On `ubuntu-latest`/GitHub-hosted or plain self-hosted runners (no Tailscale/custom routing) in `--network-isolation` mode, `awf-cli-proxy` never becomes healthy because tcp-tunnel dials `ENETUNREACH 172.17.0.1:18443` against `host.docker.internal`, exhausting the DIFC liveness probe before the agent starts. `awf-net` is `internal: true` with no outbound route. Squid and api-proxy were already dual-homed onto `awf-ext`, but `cli-proxy` was attached only to `awf-net` even though it uses `host.docker.internal` to reach the external DIFC proxy. **Known unresolved:** the dual-home attempt in github/gh-aw-firewall#7066 was reverted because attaching `cli-proxy` to `awf-ext` grants general external egress from that sidecar. A constrained host-reachability fix has not merged yet, so upgrading to #7066 does not resolve this mode.
 
+B21 / github/gh-aw-firewall#7148, github/gh-aw-firewall#7150 — Concurrent JVM-heavy builds (javac, Android manifest merger) inside the AWF agent container fail with `unable to create native thread` / `Cannot create worker GC thread`, and `/sys/fs/cgroup` inside the sandbox exposes no `pids.max`/`pids.current` (`ulimit -u` reports unlimited), so JVM tools have no way to discover or size against the real process/thread ceiling. AWF hardcoded Docker's `pids_limit` to 1000 with no visibility or configurability. **Fixed in AWF (PR github/gh-aw-firewall#7150, merged 2026-08-09):** a new `--pids-limit <n>` CLI flag (default 1000, matching prior behavior) with `container.pidsLimit` config-file support is plumbed through `cli-options.ts` → `validators/log-and-limits.ts` (`parsePidsLimit`) → `build-config.ts` → `services/agent-service.ts`. `containers/agent/entrypoint.sh` adds `mount_host_cgroupfs()` (best-effort) to bind-mount the container's delegated `/sys/fs/cgroup` read-only onto `/host/sys/fs/cgroup` so `pids.max`/`pids.current` are visible inside chroot. Upgrade AWF to include github/gh-aw-firewall#7150; raise the ceiling with `--pids-limit 4000` (or similar) for concurrent JVM builds.
+
+B22 / github/gh-aw-firewall#7149, github/gh-aw-firewall#7152, github/gh-aw-firewall#7132 — Strict-security (`--network-isolation`, no `--legacy-security`) workflows cannot directly reach a GitHub Actions `services:` container that uses a raw protocol (for example Postgres on 5432) through `--enable-host-access` / `--allow-host-ports`. Strict topology omits the agent's `host.docker.internal` mapping and host-access iptables bypass, while raw clients such as `psql` cannot use Squid's HTTP `CONNECT` protocol. **Known unresolved:** AWF needs a supported strict-topology route for raw service protocols, and gh-aw must derive the required service ports. Until then, run direct service clients outside strict topology (for example with `--legacy-security`) or use a separately verified tunnel.
+
+B23 / github/gh-aw-firewall#7130 (still open), github/gh-aw-firewall#7147, github/gh-aw-firewall#7151 — Copilot-engine workflows fail with `spawn /usr/local/bin/copilot ENOENT` specifically when the runner's tool-cache already has `copilot-cli` installed (a cache hit). This is **not an AWF defect**: gh-aw's `install_copilot_cli.sh` `activate_cached_copilot_bin()` prepends the cached dir to `PATH` and returns early on cache hits, skipping the wrapper install to `/usr/local/bin/copilot`, while the compiler-emitted harness (`copilot_harness.cjs`) always spawns the hardcoded absolute path `/usr/local/bin/copilot`. AWF's agent container mounts host `/usr`/`/usr/local` read-only, reflecting host state at container *start*, so a workaround symlink must exist on the host before `awf` is invoked. **No AWF code change (documentation only, PR github/gh-aw-firewall#7151, merged 2026-08-09)**: `docs/troubleshooting.md` gains a "Harness Binary Resolution Issues" section explaining the read-only `/usr` mount timing and that `chroot.binariesSourcePath` only helps `PATH`-based lookups, not hardcoded-absolute-path harness bugs. The durable fix belongs in gh-aw's installer/harness, tracked as github/gh-aw-firewall#7130 (open). Workaround: `sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot` on the **host**, before invoking `awf`.
+
 D8 / github/gh-aw-firewall#6401, github/gh-aw-firewall#6326 — Under `--container-runtime gvisor` or raw `runsc`, MCP calls to the gateway at `172.30.0.1:8080` could be misrouted through Squid and fail with `403 ERR_ACCESS_DENIED` because gVisor's userspace netstack does not use the host iptables DNAT bypass. **Fixed in AWF (PR github/gh-aw-firewall#6401)**: `runtimeUsesIptables()` now skips `awf-iptables-init` for `gvisor`, its `runsc` alias, and `sbx`, and the MCP gateway plus `host.docker.internal` are added to `NO_PROXY` for proxy-aware clients. Caveat: proxy-unaware raw sockets (for example `/dev/tcp`) still fail with `No route to host` under gVisor.
 
 D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`. **Fixed in AWF (PR github/gh-aw-firewall#6336)**: sbx now mounts only whitelisted home subdirectories, and `scrubHomeCredentials()` / `restoreHomeCredentials()` temporarily move nested credential files out of the mounted tree during sandbox lifetime.