Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions docs/environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -450,10 +450,11 @@ container:
pidsLimit: 4000
```

`--pids-limit` is a Docker Compose agent setting. It is unsupported by microVM
runtimes such as `--container-runtime sbx`: AWF warns and ignores it because
the sandbox does not support passing through the Docker agent cgroup or its
`pids.max`/`pids.current` metrics.
`--pids-limit` is a Docker Compose agent setting. It is unsupported by the
`--container-runtime sbx` and `--container-runtime cloud-hypervisor` microVM
runtimes, so AWF warns and ignores it rather than passing through Docker agent
cgroup metrics. Cloud Hypervisor applies its own fixed, defense-in-depth
`pids.max` limit to the VMM's dedicated cgroup v2 leaf.

## Troubleshooting

Expand Down
46 changes: 25 additions & 21 deletions docs/gvisor-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,16 @@ runtime and `--container-runtime gvisor` — to add host-kernel isolation to the
agent container. It is written for two audiences:

1. Engineers who want to understand *how the existing gVisor integration works*.
2. Ourselves, when evaluating or adding other agent-isolation runtimes (Kata,
another OCI runtime, or gVisor's own KVM platform on bare-metal runners).
2. Engineers comparing gVisor with the sbx and Cloud Hypervisor microVM
backends, or adding another agent-isolation runtime.

:::note
gVisor is the **compose-model** counterpart to the microVM backend documented in
the pending
[Docker Sandboxes (sbx) integration guide (PR #6331)](https://github.com/github/gh-aw-firewall/pull/6331).
gVisor is the **compose-model** counterpart to the
[Docker Sandboxes (sbx)](./sbx-integration.md) and
[Cloud Hypervisor](./cloud-hypervisor-foundation.md) microVM backends.
With gVisor the agent stays an ordinary Docker Compose service (just with a
hardened runtime); with sbx the agent leaves compose entirely and runs in a
microVM. See also
hardened runtime); with sbx or Cloud Hypervisor the agent leaves compose
entirely and runs in a microVM. See also
[Sandbox design](./sandbox-design.md) for why the *default* backend is plain
Docker + Squid.
:::
Expand Down Expand Up @@ -87,20 +87,21 @@ as sbx or Cloud Hypervisor, which boots a separate Linux kernel.

## Part 2 — How AWF uses gVisor

Unlike the sbx microVM backend, gVisor keeps the agent as a **normal Docker
Compose service** — AWF simply sets the service's `runtime:` to `runsc`. The
whole existing AWF model (Squid egress ACL, iptables DNAT, api-proxy credential
injection, chroot, capability drop) stays in place; gVisor adds a hardened kernel
boundary *underneath* it as defense-in-depth.
Unlike the sbx and Cloud Hypervisor microVM backends, gVisor keeps the agent as
a **normal Docker Compose service** — AWF simply sets the service's `runtime:`
to `runsc`. The whole existing AWF model (Squid egress ACL, api-proxy credential
injection, chroot, capability drop) stays in place; gVisor adds a hardened
kernel boundary *underneath* it as defense-in-depth.

### The `executionModel` abstraction (`src/container-runtime.ts`)

gVisor is registered with `executionModel: 'compose'`:

```ts
const RUNTIME_REGISTRY = {
gvisor: { executionModel: 'compose', dockerRuntime: 'runsc', needsStaticDns: true, usesIptables: false },
sbx: { executionModel: 'microvm', dockerRuntime: undefined, needsStaticDns: false, usesIptables: false },
gvisor: { executionModel: 'compose', dockerRuntime: 'runsc', needsStaticDns: true, usesIptables: false },
sbx: { executionModel: 'microvm', dockerRuntime: undefined, needsStaticDns: false, usesIptables: false },
'cloud-hypervisor': { executionModel: 'microvm', dockerRuntime: undefined, needsStaticDns: false, usesIptables: false },
};
```

Expand Down Expand Up @@ -138,7 +139,8 @@ identical to default Docker mode*: AWF applies selective bind mounts under
`chroot`s into `/host` before running:

- `/usr`, `/bin`, `/sbin`, `/lib`, `/lib64`, `/opt`, `/sys`, `/dev` → `/host/*`
read-only (system libraries and toolchains come from the host, unlike sbx).
read-only (system libraries and toolchains come from the host, unlike the sbx
and Cloud Hypervisor guest-image models).
- `<workspaceDir>` → `/host<workspaceDir>` read-write; `/tmp` → `/host/tmp`
read-write.
- When `chroot.binariesSourcePath` is configured, that tool directory is additionally
Expand All @@ -164,11 +166,12 @@ the legacy Sentry-Gofer protocol.) Practical consequences:
below and should be validated when a tool works in Docker but not under gVisor.

:::note
Because gVisor reuses the compose agent, there is **no sbx-style host-path ==
guest-path** behavior here: the agent sees files under `/host` (pre-chroot) and
at their normal paths (post-chroot), exactly as in default Docker mode. A new
compose-model runtime inherits this mount set for free; a new *microVM* runtime
(like sbx) must define its own sharing scheme instead.
Because gVisor reuses the compose agent, it does not use either microVM sharing
model: sbx preserves host paths in the guest, while Cloud Hypervisor exports the
workspace at `/workspace` through sandboxed virtio-fs. The gVisor agent sees
files under `/host` (pre-chroot) and at their normal paths (post-chroot), exactly
as in default Docker mode. A new compose-model runtime inherits this mount set
for free; a new microVM runtime must define its own sharing scheme.
:::

### The netstack DNS problem (and the fix)
Expand Down Expand Up @@ -363,5 +366,6 @@ service.
- AWF source: `src/container-runtime.ts`, `src/services/agent-service.ts`,
`src/topology.ts`, `src/services/agent-environment/tool-specific-environment.ts`
- CI: `.github/workflows/test-gvisor-compat.yml`, `.github/workflows/smoke-gvisor*.md`
- Related: [Docker Sandboxes (sbx) integration guide (PR #6331)](https://github.com/github/gh-aw-firewall/pull/6331),
- Related: [Docker Sandboxes (sbx) integration](./sbx-integration.md),
[Cloud Hypervisor architecture](./cloud-hypervisor-foundation.md),
[Sandbox design](./sandbox-design.md)
41 changes: 30 additions & 11 deletions docs/mount-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,32 +8,40 @@ drift. They are now centralized in a single declarative config:
- **Config:** [`src/config/sandbox-mount-policy.json`](../src/config/sandbox-mount-policy.json)
- **Loader / typed accessors:** [`src/config/mount-policy.ts`](../src/config/mount-policy.ts)

Every runtime reads from this one source of truth, so the Docker/runc compose
agent, the gVisor/runsc compose agent, and the sbx microVM can no longer diverge.
The Docker/runc compose agent, the gVisor/runsc compose agent, and the sbx
microVM all read from this one source of truth, so they can no longer diverge.
The Cloud Hypervisor microVM does not currently consult this policy: its
production export path (`src/cloud-hypervisor/exports.ts`) only resolves the
workspace, runner tool cache, and `RUNNER_TEMP`/`tmp/gh-aw` directories — it
does not export `$HOME` at all, so the home allow/deny lists have no
production call site for this runtime. `MicrovmWorkspaceImage`
(`src/microvm/workspace.ts`) does read `home.toolSubdirs` and
`credentials.entries`, but it has no production call site today; it is
exercised only by its own tests.

## What the policy contains

| Section | Kind | Applies to | Consumed by |
| --- | --- | --- | --- |
| `system.directories.default` / `.sysroot` | allow (dirs) | compose (Docker + gVisor) | `system-mounts.ts` |
| `system.etc` | allow (files) | compose (Docker + gVisor) | `etc-mounts.ts` |
| `home.toolSubdirs` | allow (dirs) | all runtimes | `home-strategy.ts`, `sbx-manager.ts` |
| `home.forbiddenSubdirs` | deny guard | all runtimes | invariant tests |
| `credentials.entries` | deny (files/dirs) | all runtimes | `credential-hiding.ts`, `sbx-manager.ts` |
| `home.toolSubdirs` | allow (dirs) | compose + sbx | `home-strategy.ts`, `sbx-manager.ts` |
| `home.forbiddenSubdirs` | deny guard | compose + sbx | invariant tests |
| `credentials.entries` | deny (files/dirs) | compose + sbx | `credential-hiding.ts`, `sbx-manager.ts` |

The `system.*` section is compose-only: the sbx microVM gets its system
libraries from its guest image, not from host mounts.
The `system.*` section is compose-only: sbx gets its system libraries from a
guest image, not from host mounts. Cloud Hypervisor also boots from a guest
image and is not driven by this config at all.

## How each runtime applies the credential deny list

The two backends hide credentials with different mechanisms, but from the **same
list**:
The runtime families that read the policy hide credentials with different
mechanisms, but from the **same list**:

- **Compose (Docker / gVisor)** mounts an empty `$HOME` plus the `toolSubdirs`,
then blanks each credential **file** with a `/dev/null` bind overlay
(`credential-hiding.ts`). For a `dir` entry it masks the enumerated `files`;
for a `file` entry it masks the path itself. Directory entries with no known
filenames can't be masked this way and are covered only by sbx.
for a `file` entry it masks the path itself.
- **sbx microVM** mounts the `toolSubdirs` (plus `.copilot`/`.gemini`) wholesale,
because sbx positional mounts are directory-granular and can't overlay
`/dev/null` onto a nested path. Before `sbx create` it **moves** each credential
Expand All @@ -42,6 +50,17 @@ list**:
top-level parent is actually mounted — paths under never-mounted dirs like
`.ssh` or `.aws` are skipped because they never enter the VM.

**Cloud Hypervisor microVM** does not go through this list today. Its
production export path (`src/cloud-hypervisor/exports.ts`) never exports
`$HOME` — only the workspace, runner tool cache, and `RUNNER_TEMP`/`tmp/gh-aw`
directories are exposed to the guest — so host credential dotfiles are simply
never part of the guest export, independent of `credentials.entries`.
`MicrovmWorkspaceImage` (`src/microvm/workspace.ts`) does read
`home.toolSubdirs` and exclude `credentials.entries` when staging a workspace
image, but it has no production call site; it exists only for its own test
coverage. If it is wired up for Cloud Hypervisor in the future, this doc
should be updated accordingly.

In all cases the agent receives the credentials it legitimately needs through the
API proxy or environment, never from these on-disk stores.

Expand Down
17 changes: 11 additions & 6 deletions docs/sbx-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ to run an agent inside a hypervisor-isolated microVM while keeping AWF's own
egress-filtering infrastructure on the host. It is written for two audiences:

1. Engineers who want to understand *how the existing sbx integration works*.
2. Engineers who want to add another KVM-based microVM backend, such as a
bespoke krun-based runner.
2. Engineers comparing sbx with AWF's implemented
[Cloud Hypervisor preview](./cloud-hypervisor-foundation.md), or adding
another KVM-based microVM backend.

:::note
This is distinct from [Sandbox design](./sandbox-design.md), which explains why
Expand Down Expand Up @@ -134,7 +135,8 @@ Three capability queries drive the rest of the codebase:
(`gvisor` → `runsc`); returns `undefined` for microVM backends (they don't use
Docker's `runtime:` field).
- `runtimeNeedsStaticDns(name)` — whether AWF must inject static `/etc/hosts`
entries (gVisor needs this; sbx manages its own DNS).
entries (gVisor needs this; sbx and Cloud Hypervisor manage DNS outside
Docker's embedded resolver).
- `runtimeUsesComposeAgent(name)` — **the key switch**: `false` for `microvm`
models. When false, the agent is *not* emitted into `docker-compose.yml`, and
lifecycle is driven by the microVM CLI instead of `docker logs`/`docker wait`.
Expand Down Expand Up @@ -360,14 +362,16 @@ flowchart TB
sbxproxy -->|not filtered by AWF ACL| internet
```

## Part 3 — Adding another KVM-based microVM backend
## Part 3 — Cloud Hypervisor and the external microVM seam

Because the microVM path is abstracted behind a small set of seams, adding a new
KVM backend is mostly a matter of implementing a manager and registering it.
Cloud Hypervisor is the repository's fail-closed KVM workload preview built on
this seam. It uses a dedicated network namespace and reaches Squid and the API
proxy through AWF-managed networking. The following checklist applies to other
backends.
backends. See
[Cloud Hypervisor architecture](./cloud-hypervisor-foundation.md) for its
support boundary, security model, and lifecycle.

### 1. Register the runtime

Expand Down Expand Up @@ -474,4 +478,5 @@ a way to force egress through AWF's Squid.
- AWF source: `src/container-runtime.ts`, `src/sbx-manager.ts`,
`src/commands/main-action.ts`, `src/commands/validators/security-mode.ts`,
`src/cloud-hypervisor-runtime-backend.ts` (KVM backend built on this seam)
- Related: [Sandbox design](./sandbox-design.md), [Architecture](./architecture.md)
- Related: [Cloud Hypervisor architecture](./cloud-hypervisor-foundation.md),
[Sandbox design](./sandbox-design.md), [Architecture](./architecture.md)