diff --git a/docs/arc-dind.md b/docs/arc-dind.md index b95414e76..39d63d2cf 100644 --- a/docs/arc-dind.md +++ b/docs/arc-dind.md @@ -131,7 +131,7 @@ For fine-grained control (or when not using `runner.topology`): ## Field behavior - `chroot.identity.*`: applied inside entrypoint **after** `chroot /host` to override HOME/USER/LOGNAME and identity mapping hints. -- `chroot.binariesSourcePath`: mounts a runner-side binaries directory over `/usr/local/bin` inside chroot mode so runner-installed CLIs are visible even when `/usr` comes from the DinD daemon filesystem. +- `chroot.binariesSourcePath`: mounts a runner-side binaries directory at `/host/tmp/awf-runner-bin` (inside chroot: `/tmp/awf-runner-bin`) and prepends it to `PATH`, so runner-installed CLIs are visible even when `/usr` comes from the DinD daemon filesystem. - `dind.preStageDirs`: runs a short-lived staging container in DinD mode to create required workdir tree with open permissions. - `dind.stageEngineBinary`: copies an engine binary from the runner path into daemon-visible filesystem before compose startup. - `dind.stagingImage`: image used for short-lived staging containers. diff --git a/docs/awf-config-spec.md b/docs/awf-config-spec.md index 6bad4f972..d430d7f27 100644 --- a/docs/awf-config-spec.md +++ b/docs/awf-config-spec.md @@ -188,7 +188,7 @@ AWF settings MAY be supplied via config files, including stdin (`--config -`). - `container.runnerToolCachePath` → *(config-only; checked first for optional read-only runner tool cache mount, before `RUNNER_TOOL_CACHE` and `/home/runner/work/_tool` auto-detection)* - `container.mounts[]` → `-v, --mount` *(repeatable; each array entry maps to one Docker volume mount in `/host_path:/container_path[:ro|rw]` format (both paths must be absolute; host path must exist); in chroot mode, container paths are automatically prefixed with `/host`)* - `container.containerRuntime` → `--container-runtime` *(user-facing runtime name: `"gvisor"` for OCI runtime in compose, `"sbx"` for Docker sbx microVM. For gvisor: translates to `"runsc"`, injects `extra_hosts` for DNS workaround. For sbx: agent runs in a hypervisor-isolated microVM, infra stays in compose, sbx proxy chains through AWF's Squid.)* -- `chroot.binariesSourcePath` → *(config-only; overlays a runner-side binaries directory at `/usr/local/bin` inside chroot mode)* +- `chroot.binariesSourcePath` → *(config-only; mounts a runner-side binaries directory at `/tmp/awf-runner-bin` inside chroot mode and prepends it to `PATH`)* - `chroot.identity.home` → *(config-only; forwarded as `AWF_CHROOT_IDENTITY_HOME` and applied after chroot pivot)* - `chroot.identity.user` → *(config-only; forwarded as `AWF_CHROOT_IDENTITY_USER` and applied to `USER`/`LOGNAME` after chroot pivot)* - `chroot.identity.uid` → *(config-only; forwarded as `AWF_CHROOT_IDENTITY_UID` for chroot user mapping)* diff --git a/docs/awf-config.schema.json b/docs/awf-config.schema.json index bf0f6dbee..3c4eb2d38 100644 --- a/docs/awf-config.schema.json +++ b/docs/awf-config.schema.json @@ -656,7 +656,7 @@ "properties": { "binariesSourcePath": { "type": "string", - "description": "Optional runner-side directory to overlay at /usr/local/bin inside chroot mode (for split-filesystem ARC/DinD runners)." + "description": "Optional runner-side directory mounted at /tmp/awf-runner-bin inside chroot mode and prepended to PATH (for split-filesystem ARC/DinD runners)." }, "identity": { "type": "object", diff --git a/docs/chroot-mode.md b/docs/chroot-mode.md index 6141c9e69..9083eb986 100644 --- a/docs/chroot-mode.md +++ b/docs/chroot-mode.md @@ -189,7 +189,7 @@ When `chroot.binariesSourcePath` is set in stdin config, AWF also mounts: | Host Path | Container Path | Purpose | |-----------|----------------|---------| -| `chroot.binariesSourcePath` | `/host/usr/local/bin:ro` | Overlay runner-installed binaries in chroot PATH | +| `chroot.binariesSourcePath` | `/host/tmp/awf-runner-bin:ro` | Overlay runner-installed binaries in chroot PATH (prepended as `/tmp/awf-runner-bin`) | **Note:** As of v0.13.13, `/proc` is no longer bind-mounted. Instead, a fresh container-scoped procfs is mounted at `/host/proc` during entrypoint initialization. This provides dynamic `/proc/self/exe` resolution required by Java and .NET runtimes. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9779b1469..063ec8931 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -301,6 +301,58 @@ AWF uses a forward proxy (Squid) for HTTPS egress control rather than transparen - **Java tools**: Use `JAVA_TOOL_OPTIONS` with JVM system properties (set automatically by AWF) - **Maven**: Requires `~/.m2/settings.xml` (must be configured manually — see above) +## Harness Binary Resolution Issues + +### `spawn /usr/local/bin/ ENOENT` (hardcoded absolute paths) + +**Problem:** An agentic harness (e.g. the gh-aw Copilot engine) fails with an error like: + +``` +spawn /usr/local/bin/copilot ENOENT +``` + +even though the tool is installed and resolvable via `PATH` on the runner. + +**Cause:** Some harnesses hardcode an absolute path to the tool binary (e.g. +`/usr/local/bin/copilot`) instead of doing a `PATH` lookup, and their +installer/cache-hit logic can skip creating that file (e.g. a tool-cache hit +short-circuits before the `/usr/local/bin` wrapper is installed). This is a +bug in the harness/installer, not in AWF — but it interacts with how AWF's +agent container mounts the host filesystem: + +- AWF's agent container mounts host `/usr` (and therefore `/usr/local`) + **read-only** at `/host/usr` (chroot mode) so it can't be written to from + *inside* the container. +- The bind mount is a live view of host `/usr/local/bin`: changes made on the + **host** (including host-side `pre-agent-steps`) are visible in the sandbox. +- Commands that run *inside* the AWF sandbox still cannot create this symlink, + because `/usr` is mounted read-only there. + +**Workarounds:** + +1. **Host-side symlink before invoking `awf`** — if you control the step + immediately before AWF starts the sandbox, create the missing binary on + the *host* filesystem so it is present when AWF takes its `/usr` bind + mount: + ```bash + sudo ln -sf "$(command -v copilot)" /usr/local/bin/copilot + sudo awf --allow-domains ... -- + ``` + Doing this from *inside* the sandboxed command will not work, since + `/usr/local` is read-only once the container is running. +2. **`chroot.binariesSourcePath`** — point this config option at a host + directory containing the tool binary (or a symlink to it). AWF mounts it + read-only at `/host/tmp/awf-runner-bin` and `entrypoint.sh` prepends it to + the chrooted `PATH`. This fixes `PATH`-based lookups, but does **not** + help harnesses that spawn a hardcoded absolute path (like + `/usr/local/bin/copilot`) rather than resolving the binary via `PATH`. See + [docs/awf-config-spec.md](awf-config-spec.md) §`chroot.binariesSourcePath`. +3. **Fix upstream** — the durable fix is in the harness/installer itself + (e.g. gh-aw's `install_copilot_cli.sh` / `pkg/constants/constants.go`), so + that it always resolves the binary via `PATH` or always populates the + hardcoded path, even on a tool-cache hit. Track/coordinate with the + upstream project for the permanent fix. + ## Log Analysis ### Finding Blocked Domains diff --git a/src/awf-config-schema.json b/src/awf-config-schema.json index bf0f6dbee..3c4eb2d38 100644 --- a/src/awf-config-schema.json +++ b/src/awf-config-schema.json @@ -656,7 +656,7 @@ "properties": { "binariesSourcePath": { "type": "string", - "description": "Optional runner-side directory to overlay at /usr/local/bin inside chroot mode (for split-filesystem ARC/DinD runners)." + "description": "Optional runner-side directory mounted at /tmp/awf-runner-bin inside chroot mode and prepended to PATH (for split-filesystem ARC/DinD runners)." }, "identity": { "type": "object",