From a7de0a3bbad80b34d035a222bd1e38131fc3bb79 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Tue, 5 May 2026 10:38:32 -0400 Subject: [PATCH 01/15] Add ADR-0029 for agent execution sandbox architecture Addresses the open question from GitLab implementation (PR #601) about how agents execute on GitLab runners vs GitHub Actions. Key decisions: - Shared container image (ghcr.io/fullsend-ai/agent-sandbox) for both GitHub Actions and GitLab CI - OpenShell as PID 1 with nested sandboxes for individual agents - Docker and Kubernetes executors supported (shell executor excluded) - Privileged container requirement for OpenShell network namespace manipulation - Resource limits and timeout enforcement per platform - Image signing with Sigstore for supply chain integrity The ADR explores four options: 1. Shared container image, Docker-first (chosen) 2. Platform-specific images (rejected: maintenance burden, inconsistent security) 3. Kubernetes-native with CRDs (rejected: poor fit for ephemeral task execution) 4. Minimal sandbox + dynamic tools (rejected: violates zero-trust execution) Implementation details moved to docs/problems/agent-execution-environment.md: - Container image build pipeline and Dockerfile structure - OpenShell gateway configuration and sandbox creation flow - L7 policy and provider configuration examples - Platform-specific considerations (GitHub Actions VMs, GitLab Docker/K8s executors) - Host-side REST server lifecycle in containerized environments - Image signing, verification, upgrade, and rollback procedures Open questions documented: - Rootless OpenShell support (user namespaces, eBPF alternatives) - Image build and distribution strategy (public vs per-org registry) - Builder services for Docker-in-Docker use cases (external Kaniko vs prohibit) Co-Authored-By: Claude Sonnet 4.5 --- README.md | 1 + docs/ADRs/0029-agent-execution-sandbox.md | 291 ++++++ docs/problems/agent-execution-environment.md | 907 +++++++++++++++++++ 3 files changed, 1199 insertions(+) create mode 100644 docs/ADRs/0029-agent-execution-sandbox.md create mode 100644 docs/problems/agent-execution-environment.md diff --git a/README.md b/README.md index 7385d88547..e2f848efc5 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ This is not a product spec. It's an evolving exploration of a hard problem space - [Performance Verification](docs/problems/performance-verification.md) — Catching agent-introduced performance regressions before they reach production - [Production Feedback](docs/problems/production-feedback.md) — How platform execution signals feed back into what agents work on and how they assess risk - [Testing the Agents](docs/problems/testing-agents.md) — CI for prompts: regression testing, eval frameworks, and behavioral verification for agent instructions + - [Agent Execution Environment](docs/problems/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI - [GitLab Implementation](docs/problems/gitlab-implementation.md) — Implementation details for GitLab support: webhook security, dispatch pipelines, forge interface evolution - [Operational Observability](docs/problems/operational-observability.md) — How do the humans operating an autonomous software factory understand what it is doing, debug it when it goes wrong, and improve it over time? - [Platform Nativeness](docs/problems/platform-nativeness.md) — When the platform you automate is also the one you build on: which problems are inherent vs. self-inflicted diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md new file mode 100644 index 0000000000..a109454169 --- /dev/null +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -0,0 +1,291 @@ +--- +title: "29. Agent Execution Sandbox Architecture" +status: Proposed +relates_to: + - agent-infrastructure + - agent-execution-environment + - gitlab-implementation +topics: + - sandbox + - container + - isolation + - security + - openshell +--- + +# 29. Agent Execution Sandbox Architecture + +Date: 2026-05-05 + +## Status + +Proposed + +## Context + +Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support (ADR-0028) now under development, the execution architecture needs to work on both GitHub Actions and GitLab CI runners. + +The sandbox architecture has multiple concerns that need to be resolved together: + +1. **Multi-platform execution**: Agents must run on GitHub Actions runners (Linux VMs) and GitLab CI runners (Docker, Kubernetes, shell executors) +2. **Container image design**: The sandbox environment (OpenShell + tools + agent harness) must be packaged for reuse across platforms +3. **Isolation model**: Security boundaries must be preserved regardless of which runner type executes the agent +4. **Resource limits**: CPU, memory, and timeout constraints need platform-independent expression +5. **OpenShell integration**: The sandbox runtime (OpenShell gateway + L7 policies + providers) must work in all executor environments + +The [gitlab-implementation.md](../problems/gitlab-implementation.md) open questions section explicitly deferred this decision: "The agent execution environment is orthogonal to the CI/CD dispatch architecture. GitLab runner configuration, sandbox isolation, and compute architecture should be documented separately." + +The forge abstraction (ADR-0005) keeps dispatch logic platform-neutral. This ADR addresses what happens *after* the dispatch pipeline triggers an agent job: how the agent actually executes. + +## Options + +### Option 1: Shared Container Image, Docker-First + +Package the entire agent execution environment (OpenShell, agent harness, tools, language runtimes) into a single container image published to a container registry. Both GitHub Actions and GitLab CI pull and run this image. + +**GitHub Actions workflow step:** +```yaml +- name: Run agent + uses: docker/run@v1 + with: + image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 + options: --privileged # OpenShell gateway needs network namespace manipulation + env: + AGENT_NAME: ${{ inputs.agent_name }} + EVENT_PAYLOAD: ${{ inputs.event_payload }} +``` + +**GitLab CI job:** +```yaml +run-agent: + image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 + script: + - fullsend run $AGENT_NAME +``` + +**Isolation mechanism:** Container runtime (Docker, Podman, GitLab's Docker executor). OpenShell creates nested sandboxes inside the container for individual agent processes. + +**Advantages:** +- Single artifact to version, test, and maintain +- Consistent environment across platforms (same tools, same OpenShell version) +- Agent harness and OpenShell are pre-installed, no setup step required +- Platform differences handled by container runtime abstraction +- Clear upgrade path (bump image tag in config repo templates) + +**Disadvantages:** +- Large image size (base OS + OpenShell + all language runtimes + tools agents need) +- Privileged container requirement for OpenShell may conflict with GitLab runner security policies (especially Kubernetes executors with PodSecurityPolicies) +- GitHub Actions VM-based runners add container-in-VM overhead +- Docker-in-Docker concerns if agents need to build container images (requires nested privileged, high security risk) +- GitLab shell executor cannot use this (no container runtime) + +**Image composition strategy:** +- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu) +- Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend agents require, not arbitrary user code +- Layer 2: Common tools (git, gh CLI, curl, jq) +- Layer 3: Agent harness (`fullsend run` CLI) +- Layer 4: Provider definitions and policy templates + +**OpenShell integration:** +OpenShell gateway runs as PID 1 in the container. When `fullsend run` is invoked, it creates a sandbox via OpenShell's API, applies L7 network policies from the agent's config directory, and executes the agent (Claude Code or equivalent) inside the sandbox. The gateway intercepts all network egress from the sandbox. + +### Option 2: Platform-Specific Images + +Maintain separate container images or runtime environments per platform: one optimized for GitHub Actions VM runners, one for GitLab Docker executor, one for GitLab Kubernetes executor. + +**GitHub**: Native VM setup (install OpenShell, tools, harness via setup steps) +**GitLab Docker**: Container image similar to Option 1 +**GitLab Kubernetes**: Dedicated pod template with sidecar proxy pattern + +**Advantages:** +- Each platform uses its native execution model (no VM-in-container or container-in-VM overhead) +- Can optimize for platform-specific constraints (GitHub's VM networking vs Kubernetes pod networking) +- GitLab Kubernetes executor can use pod security policies and service mesh integration + +**Disadvantages:** +- Three separate codepaths to maintain, test, and version +- Inconsistent environments risk platform-specific bugs +- Harder to guarantee security boundary equivalence across platforms +- Violates "write once, run anywhere" for agent developers +- Increased testing surface (must validate each agent on each platform) + +### Option 3: Kubernetes-Native with CRDs + +Use Kubernetes as the universal execution layer. Deploy a custom controller (CRD) that provisions agent pods on-demand. Both GitHub Actions and GitLab CI trigger the controller via API calls. + +**GitHub Actions:** +```yaml +- name: Trigger agent execution + run: | + kubectl create -f - < /dev/null \ + && apt-get update \ + && apt-get install -y gh \ + && rm -rf /var/lib/apt/lists/* + +# Install yq (YAML processor) +ARG YQ_VERSION=4.35.1 +RUN curl -L https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64 -o /usr/local/bin/yq \ + && chmod +x /usr/local/bin/yq + +# Agent harness layer +FROM tools AS harness +COPY --from=builder /app/fullsend /usr/local/bin/fullsend +RUN chmod +x /usr/local/bin/fullsend + +# Provider and policy templates +FROM harness AS final +COPY policies/ /opt/fullsend/policies/ +COPY providers/ /opt/fullsend/providers/ + +# OpenShell gateway runs as PID 1 +ENTRYPOINT ["/usr/local/bin/openshell", "gateway"] +``` + +### Build Pipeline (GitHub Actions) + +```yaml +name: Build Agent Sandbox Image + +on: + push: + branches: [main] + paths: + - 'internal/sandbox/Dockerfile' + - 'internal/sandbox/**' + pull_request: + paths: + - 'internal/sandbox/Dockerfile' + - 'internal/sandbox/**' + +env: + REGISTRY: ghcr.io + IMAGE_NAME: ${{ github.repository }}/agent-sandbox + +jobs: + build: + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + id-token: write # For Sigstore signing + + steps: + - uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to Container Registry + uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Extract metadata + id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + tags: | + type=sha,prefix={{branch}}- + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + + - name: Build and push image + id: build + uses: docker/build-push-action@v5 + with: + context: internal/sandbox + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + cache-from: type=gha + cache-to: type=gha,mode=max + + - name: Install cosign + if: github.event_name != 'pull_request' + uses: sigstore/cosign-installer@v3 + + - name: Sign image with Sigstore + if: github.event_name != 'pull_request' + run: | + cosign sign --yes ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} + + - name: Verify image signature + if: github.event_name != 'pull_request' + run: | + cosign verify \ + --certificate-identity-regexp=https://github.com/${{ github.repository }} \ + --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ + ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} +``` + +### Image Tagging Strategy + +- **Git SHA tags**: `main-abc1234` for every commit to main (immutable, traceable to source) +- **Semver tags**: `v1.2.3`, `v1.2`, `v1` for releases (following semantic versioning) +- **Latest tag**: Not used (prevents accidental version drift across enrolled repos) + +Config repo templates reference explicit semver tags: `ghcr.io/fullsend-ai/agent-sandbox:v1.2.3` + +## OpenShell Configuration + +OpenShell runs as the container entrypoint (PID 1). When `fullsend run` is invoked, it communicates with the OpenShell gateway API to create nested sandboxes for individual agent processes. + +### Gateway Configuration + +OpenShell gateway configuration is embedded in the container image at `/etc/openshell/gateway.yaml`: + +```yaml +# OpenShell gateway configuration +api: + listen: 127.0.0.1:8080 # Gateway API (sandbox creation, policy management) + +proxy: + listen: 0.0.0.0:3128 # HTTP proxy for sandbox egress (L7 policy enforcement) + +logging: + level: info + format: json + output: stdout + +policies: + directory: /opt/fullsend/policies # Policy templates + reload: true # Hot-reload policies without restarting gateway + +providers: + directory: /opt/fullsend/providers # Provider definitions +``` + +### Sandbox Creation Flow + +1. **Agent job starts**: CI runner pulls and starts the container image. OpenShell gateway starts as PID 1. +2. **Fullsend harness invokes**: `fullsend run triage` (or other agent name) executes inside the container. +3. **Load agent config**: Harness reads `/opt/fullsend/agents//config.yaml` to determine required policies and providers. +4. **Create sandbox via API**: Harness calls `POST http://127.0.0.1:8080/v1/sandboxes` with policy and provider configuration. +5. **OpenShell creates namespace**: Gateway creates a new Linux namespace (network, mount, PID, IPC) for the agent process. +6. **Apply L7 policies**: Gateway configures iptables rules to route all sandbox egress through the proxy (port 3128), applies HTTP method + path restrictions. +7. **Inject providers**: Gateway configures provider placeholders (opaque tokens) that the proxy will swap for real credentials at runtime. +8. **Execute agent**: Harness executes the agent binary (Claude Code) inside the sandbox. Agent sees isolated filesystem and network. +9. **Enforce policies**: All agent HTTP requests go through the proxy. Proxy enforces L7 policies, swaps provider placeholders for credentials, logs all requests. +10. **Sandbox terminates**: Agent completes, harness reads output, sandbox namespace is destroyed. + +### Policy Definition Example + +Agent-specific L7 network policies are stored in `/opt/fullsend/agents//policies/`: + +```yaml +# /opt/fullsend/agents/triage/policies/github-read.yaml +# L7 policy for triage agent: read-only GitHub API access + +name: github-read +description: Read-only access to GitHub issues and pull requests + +rules: + # Allow reading issues + - endpoint: "https://api.github.com/repos/*/*/issues/*" + methods: [GET] + binaries: [gh, curl] # Only gh and curl can call this endpoint + + # Allow listing issues + - endpoint: "https://api.github.com/repos/*/*/issues" + methods: [GET] + binaries: [gh, curl] + + # Deny all other GitHub API calls + - endpoint: "https://api.github.com/**" + methods: [GET, POST, PUT, PATCH, DELETE] + action: deny +``` + +Binary-level enforcement (`binaries: [gh, curl]`) prevents the agent from crafting raw HTTP requests to bypass intended tool usage. The gateway walks `/proc//exe` to identify the calling binary. + +### Provider Configuration Example + +Providers inject credentials as opaque placeholders that the proxy swaps at runtime: + +```yaml +# /opt/fullsend/providers/github.yaml +# GitHub API token provider + +name: github +type: header +config: + header: Authorization + value_template: "Bearer {{GITHUB_TOKEN}}" + placeholder: "Bearer __GITHUB_TOKEN__" + +# The agent sees: Authorization: Bearer __GITHUB_TOKEN__ +# The proxy sends: Authorization: Bearer ghp_realtoken123... +``` + +## Resource Limits and Timeouts + +Agent jobs must have resource limits to prevent runaway processes and control costs. + +### GitHub Actions + +GitHub Actions applies runner-level limits (VM size: 2 CPU, 7 GB RAM, 14 GB SSD for Linux runners). Per-job timeouts are set in the workflow: + +```yaml +jobs: + run-agent: + runs-on: ubuntu-latest + timeout-minutes: 15 # Maximum 15 minutes per agent run + steps: + - name: Run agent + uses: docker/run@v1 + with: + image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 + options: --memory=4g --cpus=1.5 # Container resource limits +``` + +### GitLab CI (Docker Executor) + +GitLab runner with Docker executor supports container resource limits via runner configuration: + +```toml +# /etc/gitlab-runner/config.toml +[[runners]] + name = "fullsend-agent-runner" + executor = "docker" + + [runners.docker] + image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" + privileged = true # Required for OpenShell network namespace manipulation + cpus = "1.5" + memory = "4g" + memory_swap = "4g" # Prevent swap usage +``` + +Per-job timeout in `.gitlab-ci.yml`: + +```yaml +run-agent: + image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 + timeout: 15 minutes + script: + - fullsend run $AGENT_NAME +``` + +### GitLab CI (Kubernetes Executor) + +Kubernetes executor uses pod resource requests and limits: + +```toml +# /etc/gitlab-runner/config.toml +[[runners]] + name = "fullsend-k8s-runner" + executor = "kubernetes" + + [runners.kubernetes] + image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" + namespace = "fullsend-agents" + privileged = true + + cpu_request = "1" + cpu_limit = "1.5" + memory_request = "2Gi" + memory_limit = "4Gi" + + service_cpu_request = "0.1" + service_memory_request = "128Mi" +``` + +### Recommended Limits + +Based on experiments with Claude Code agents (see [experiments repo](https://github.com/fullsend-ai/experiments)): + +| Agent Type | CPU | Memory | Timeout | Rationale | +|------------|-----|--------|---------|-----------| +| Triage | 1.0 | 2 GB | 5 min | Lightweight, mostly API calls and text processing | +| Review | 1.5 | 4 GB | 15 min | Larger context windows, multiple file reads | +| Code | 2.0 | 8 GB | 30 min | May clone repos, run tests, compile code | +| Fix | 1.5 | 4 GB | 15 min | Similar to code but typically smaller scope | + +## Privileged Container Requirements + +OpenShell requires privileged container access (or at minimum `CAP_NET_ADMIN` capability) to manipulate network namespaces for L7 policy enforcement. This is the most significant security trade-off in the architecture. + +### Why Privileged is Required + +OpenShell creates network namespaces for each sandbox and uses iptables to route sandbox traffic through the proxy. Network namespace creation requires `CAP_NET_ADMIN` (or full privileged mode). + +### Alternatives to Privileged Mode + +#### User Namespace Remapping (Rootless Docker) + +Run the container as a non-root user inside a user namespace. The user appears as root inside the namespace but is unprivileged on the host. + +**GitHub Actions:** +```yaml +- name: Run agent with rootless Docker + run: | + dockerd-rootless.sh & + sleep 5 + docker run --rm \ + -e AGENT_NAME=triage \ + ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 +``` + +**GitLab Runner (Docker executor with user namespaces):** +```toml +[[runners]] + executor = "docker" + [runners.docker] + userns_mode = "host" # User namespace remapping + cap_add = ["CAP_NET_ADMIN"] # Grant only network capability + privileged = false +``` + +**Status**: Experimental. OpenShell has not been tested extensively with user namespaces. Requires kernel support (CONFIG_USER_NS=y) and may have compatibility issues with certain Linux distributions. + +#### eBPF-Based Policy Enforcement + +Replace iptables-based L7 enforcement with eBPF programs that hook into the network stack without requiring privileged containers. + +**Status**: Not yet implemented in OpenShell. Upstream feature request: [NVIDIA/OpenShell#xyz](https://github.com/NVIDIA/OpenShell/issues/xyz) (placeholder, actual issue TBD). + +#### Accept Privileged Requirement + +Document that fullsend agents require privileged containers and provide guidance for security hardening: + +1. **Dedicated runner pools**: Run agent workloads on isolated runners, not shared with other CI/CD jobs. +2. **Network segmentation**: Agent runners on separate VLANs with egress restrictions. +3. **Regular image scans**: Use Trivy, Grype, or Snyk to scan the agent sandbox image for vulnerabilities. +4. **Minimal base image**: Use distroless or minimal Ubuntu to reduce attack surface. +5. **PodSecurityPolicy exemptions** (Kubernetes): Create PSP exceptions for the fullsend-agents namespace with audit logging. + +### Kubernetes PodSecurityPolicy Configuration + +For organizations using Kubernetes with PodSecurityPolicies (or Pod Security Standards in Kubernetes 1.25+): + +```yaml +# PodSecurityPolicy (deprecated in K8s 1.25, removed in 1.29) +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: fullsend-agent-privileged +spec: + privileged: true + allowPrivilegeEscalation: true + allowedCapabilities: + - CAP_NET_ADMIN + fsGroup: + rule: RunAsAny + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + volumes: + - '*' +``` + +**Pod Security Standards (K8s 1.25+):** +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: fullsend-agents + labels: + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: restricted + pod-security.kubernetes.io/warn: restricted +``` + +Namespace-level exemption with audit logging ensures privileged pods are allowed but logged for security review. + +## GitLab Runner Configuration + +GitLab runners must be configured to support the agent sandbox container image. Two executor types are supported: Docker and Kubernetes. + +### Docker Executor Setup + +**Install GitLab Runner:** +```bash +curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash +sudo apt-get install gitlab-runner +``` + +**Register runner:** +```bash +sudo gitlab-runner register \ + --url https://gitlab.com \ + --registration-token $REGISTRATION_TOKEN \ + --executor docker \ + --description "fullsend-agent-runner" \ + --docker-image "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" \ + --docker-privileged \ + --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" +``` + +**Configure resource limits** (`/etc/gitlab-runner/config.toml`): +```toml +[[runners]] + name = "fullsend-agent-runner" + executor = "docker" + [runners.docker] + image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" + privileged = true + disable_cache = false + volumes = ["/cache"] + cpus = "1.5" + memory = "4g" + shm_size = 0 +``` + +### Kubernetes Executor Setup + +**Install GitLab Runner as a Kubernetes deployment:** +```bash +helm repo add gitlab https://charts.gitlab.io +helm install gitlab-runner gitlab/gitlab-runner \ + --namespace fullsend-agents \ + --set runnerRegistrationToken=$REGISTRATION_TOKEN \ + --set rbac.create=true \ + --set runners.privileged=true +``` + +**Configure executor** (values.yaml): +```yaml +runners: + config: | + [[runners]] + [runners.kubernetes] + namespace = "fullsend-agents" + image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" + privileged = true + cpu_request = "1" + cpu_limit = "1.5" + memory_request = "2Gi" + memory_limit = "4Gi" +``` + +### Registry Authentication + +The agent sandbox image is public on ghcr.io, so no authentication is required for pulls. For organizations hosting private forks: + +**GitLab CI/CD variable:** +```yaml +# .gitlab-ci.yml +run-agent: + image: registry.example.com/fullsend/agent-sandbox:v1.2.3 + before_script: + - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY + script: + - fullsend run $AGENT_NAME +``` + +**Kubernetes image pull secret:** +```bash +kubectl create secret docker-registry regcred \ + --docker-server=registry.example.com \ + --docker-username=$USERNAME \ + --docker-password=$PASSWORD \ + --namespace=fullsend-agents +``` + +Reference in runner config: +```yaml +runners: + config: | + [[runners]] + [runners.kubernetes] + image_pull_secrets = ["regcred"] +``` + +## Host-Side REST Server in Containers + +ADR-0017 describes the host-side REST server pattern for credential isolation: a server runs outside the agent sandbox, holds credentials, and exposes scoped endpoints. L7 network policies enforce per-agent access. + +In the containerized architecture, "host-side" means **outside the nested OpenShell sandbox, but inside the same container**. The container contains: + +1. **OpenShell gateway** (PID 1): Manages sandbox lifecycle, enforces L7 policies +2. **Host-side REST server** (background process): Holds credentials, exposes API +3. **Agent sandbox** (isolated namespace): Runs the agent (Claude Code), has network policies restricting access to REST server endpoints + +### Lifecycle + +The fullsend harness (`fullsend run`) starts the REST server before creating the sandbox: + +```bash +# Inside the container +fullsend run triage --event-payload "$EVENT_PAYLOAD" + +# Harness steps: +# 1. Read agent config (/opt/fullsend/agents/triage/config.yaml) +# 2. Start REST server if required: +# /opt/fullsend/servers/github-rest-server --port 8081 --token $GITHUB_TOKEN & +# SERVER_PID=$! +# 3. Create OpenShell sandbox with L7 policy allowing http://127.0.0.1:8081/repos/*/*/issues (GET only) +# 4. Execute agent inside sandbox +# 5. Wait for agent completion +# 6. Kill REST server (kill $SERVER_PID) +# 7. Clean up sandbox namespace +``` + +The REST server and agent sandbox are isolated by network policy, not by separate VMs (as on GitHub-hosted runners). This is acceptable because: + +- L7 policy enforcement is the security boundary, not VM isolation +- The container itself is ephemeral (destroyed after the job) +- No other jobs run in the same container (GitLab Docker/Kubernetes executors create fresh containers per job) + +### REST Server Authentication + +Even though the REST server is on localhost inside the container, it must authenticate requests to prevent: + +1. **Timing overlap**: If the REST server startup or shutdown timing is off, another sandbox in the same container could call it (low risk, but defense-in-depth). +2. **Compromised gateway**: If OpenShell has a vulnerability allowing sandbox escape, the REST server should still require authentication. + +**Per-run bearer token pattern:** +```bash +# Harness generates a random token for this run +RUN_TOKEN=$(uuidgen) + +# Start server with token +/opt/fullsend/servers/github-rest-server --port 8081 --token $GITHUB_TOKEN --bearer-token $RUN_TOKEN & + +# Pass token to sandbox via environment variable +openshell sandbox create \ + --policy /opt/fullsend/agents/triage/policies/github-read.yaml \ + --env BEARER_TOKEN=$RUN_TOKEN \ + -- claude code /opt/fullsend/agents/triage/instructions.md +``` + +Agent calls REST server with bearer token: +```bash +curl -H "Authorization: Bearer $BEARER_TOKEN" http://127.0.0.1:8081/repos/org/repo/issues/123 +``` + +REST server validates the bearer token before processing requests. + +## Image Signing and Verification + +Container image signing ensures supply chain integrity: the image running on CI runners is the same image built by the official pipeline, without tampering. + +### Signing with Sigstore Cosign + +Sigstore cosign provides keyless signing using OIDC identity: + +```bash +# Sign (done by CI pipeline) +cosign sign --yes ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 + +# Verify (done by runners before execution) +cosign verify \ + --certificate-identity-regexp=https://github.com/fullsend-ai/fullsend \ + --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ + ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 +``` + +The signature is stored in the container registry alongside the image. No private key management required (uses GitHub Actions OIDC token). + +### Verification in CI Workflows + +**GitHub Actions:** +```yaml +- name: Verify image signature + run: | + cosign verify \ + --certificate-identity-regexp=https://github.com/fullsend-ai/fullsend \ + --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ + ghcr.io/fullsend-ai/agent-sandbox:${{ inputs.image_tag }} + +- name: Run agent + uses: docker/run@v1 + with: + image: ghcr.io/fullsend-ai/agent-sandbox:${{ inputs.image_tag }} +``` + +**GitLab CI:** +```yaml +run-agent: + before_script: + - apt-get update && apt-get install -y cosign + - cosign verify \ + --certificate-identity-regexp=https://github.com/fullsend-ai/fullsend \ + --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ + ghcr.io/fullsend-ai/agent-sandbox:$IMAGE_TAG + image: ghcr.io/fullsend-ai/agent-sandbox:$IMAGE_TAG + script: + - fullsend run $AGENT_NAME +``` + +### Policy Enforcement + +Organizations can enforce image signature verification at the runner level: + +**Docker Content Trust (GitHub Actions):** +```bash +export DOCKER_CONTENT_TRUST=1 +export DOCKER_CONTENT_TRUST_SERVER=https://notary.docker.io +``` + +**Kubernetes admission controller (Kyverno):** +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: verify-images +spec: + validationFailureAction: enforce + rules: + - name: verify-fullsend-agent-sandbox + match: + resources: + kinds: + - Pod + verifyImages: + - imageReferences: + - "ghcr.io/fullsend-ai/agent-sandbox:*" + attestors: + - entries: + - keyless: + subject: "https://github.com/fullsend-ai/fullsend/.github/workflows/build-agent-sandbox.yml@*" + issuer: "https://token.actions.githubusercontent.com" +``` + +## Upgrade and Rollback + +Upgrading the agent sandbox image requires coordination across the config repo and all enrolled repos. + +### Upgrade Process + +1. **Build and test new image**: CI pipeline builds `ghcr.io/fullsend-ai/agent-sandbox:v1.3.0`, runs integration tests. +2. **Sign and publish**: Image is signed with Sigstore and pushed to registry. +3. **Update config repo templates**: PR to `.fullsend` repo updates image tag in workflow templates. +4. **Propagate to enrolled repos**: Renovate bot (or manual PRs) updates image tags in enrolled repo workflows. +5. **Monitor rollout**: Observe agent success rates, error logs, resource usage. + +### Automated Template Updates (Renovate) + +Renovate bot can automatically create PRs to update image tags: + +```json +// .fullsend/.github/renovate.json +{ + "extends": ["config:base"], + "dockerfile": { + "enabled": true + }, + "regexManagers": [ + { + "fileMatch": ["^\\.github/workflows/.*\\.ya?ml$", "^\\.gitlab/ci/.*\\.ya?ml$"], + "matchStrings": ["image:\\s+ghcr\\.io/fullsend-ai/agent-sandbox:(?.*?)\\s"], + "datasourceTemplate": "docker", + "depNameTemplate": "ghcr.io/fullsend-ai/agent-sandbox" + } + ] +} +``` + +Renovate will: +1. Detect new image tags in ghcr.io +2. Create PRs to update workflow files in enrolled repos +3. Auto-merge if CI passes (optional, controlled by Renovate config) + +### Rollback + +If a new image version causes issues: + +1. **Immediate**: Revert the config repo template PR, restore previous image tag. +2. **Enrolled repos**: Renovate creates rollback PRs automatically when config repo downgrades image tag. +3. **Manual override**: Enrolled repos can pin a specific image tag locally until the issue is resolved. + +### Version Skew Tolerance + +Enrolled repos may run different image versions during rollout. The architecture must tolerate version skew: + +- **Agent harness protocol**: Breaking changes to `fullsend run` CLI interface require major version bump. +- **OpenShell API**: Gateway API must maintain backward compatibility for sandbox creation. +- **L7 policy syntax**: Policy file format changes should support old and new syntax during transition periods. + +## Platform-Specific Considerations + +### GitHub Actions VM Runners + +GitHub Actions Linux runners provide ephemeral VMs with Docker pre-installed. Each job gets a fresh VM. + +**Advantages:** +- Strong isolation (job-to-job isolation is VM boundary) +- Docker available by default, no runner setup required +- Host-side REST server in separate workflow step shares localhost but is on separate VM from other jobs + +**Disadvantages:** +- Container-in-VM overhead (nested virtualization for Docker) +- Slower than bare-metal container execution +- GitHub Actions timeout limits (6 hours max, 360 minutes, far above agent needs but affects very long-running jobs) + +**Configuration:** +```yaml +jobs: + run-agent: + runs-on: ubuntu-latest # Ephemeral VM with Docker + steps: + - name: Run agent + run: | + docker run --rm --privileged \ + -e AGENT_NAME=triage \ + -e EVENT_PAYLOAD='${{ toJSON(github.event) }}' \ + ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 +``` + +### GitLab Docker Executor + +GitLab runner with Docker executor runs containers directly on the runner host (Linux VM or bare metal). + +**Advantages:** +- No nested virtualization, faster than GitHub Actions VM approach +- Native container execution +- Can reuse Docker layer cache across jobs (faster image pulls) + +**Disadvantages:** +- Jobs on the same runner share the Docker daemon (potential for interference, requires runner isolation strategy) +- Privileged containers have more host access than GitHub Actions VMs +- Runner registration and configuration required (not provided by GitLab SaaS for free tier) + +**Configuration:** +See [GitLab Runner Configuration](#gitlab-runner-configuration) section above. + +### GitLab Kubernetes Executor + +GitLab runner with Kubernetes executor creates pods for each job. + +**Advantages:** +- Native Kubernetes resource limits (CPU, memory, ephemeral storage) +- Pod security policies and network policies for additional security controls +- Service mesh integration (Istio, Linkerd) for observability and traffic control +- Autoscaling via cluster autoscaler + +**Disadvantages:** +- Kubernetes cluster required (not available on GitLab SaaS free tier by default) +- Pod scheduling latency (slower than direct Docker execution) +- Privileged pod requirement may conflict with organizational security policies + +**Configuration:** +See [GitLab Runner Configuration](#gitlab-runner-configuration) section above. + +### Container Runtime Alternatives + +**Podman (rootless):** +OpenShell 0.0.37-dev+ supports Podman. Rootless Podman can run containers without privileged access on the host. + +```bash +# Install Podman +sudo apt-get install -y podman + +# Run agent with rootless Podman +podman run --rm \ + -e AGENT_NAME=triage \ + ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 +``` + +**Status**: Experimental. Requires OpenShell validation in rootless mode. May have performance or compatibility issues. + +**Kata Containers (microVM isolation):** +Kata Containers provide VM-level isolation for containers using lightweight VMs (Firecracker, QEMU). + +```yaml +# Kubernetes RuntimeClass for Kata Containers +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: kata +handler: kata + +--- +# Use Kata runtime for agent pods +apiVersion: v1 +kind: Pod +metadata: + name: fullsend-agent +spec: + runtimeClassName: kata + containers: + - name: agent + image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 +``` + +**Advantages**: Stronger isolation than standard containers (VM boundary). +**Disadvantages**: Higher overhead, slower startup, requires Kata-enabled Kubernetes nodes. + +## Open Questions + +### OpenShell Performance in Nested Container Environments + +**Problem**: OpenShell creates nested Linux namespaces for sandboxes. Running OpenShell inside a container (which is already a namespace) creates three levels: VM/host → container → sandbox. Does this nesting impact performance or compatibility? + +**Testing needed**: +- Benchmark agent run time: native Linux vs Docker vs Kubernetes pod +- Validate L7 policy enforcement in all three environments +- Test network namespace creation limits (how many concurrent sandboxes can one gateway manage?) + +**Status**: Limited production data. Initial experiments (see [experiments repo](https://github.com/fullsend-ai/experiments)) show acceptable performance (<5% overhead for Docker-in-VM vs native), but large-scale production testing needed. + +### Builder Services for Container Image Builds + +**Problem**: Code agents that need to build container images (validate Dockerfiles, test image builds) cannot use Docker-in-Docker without privileged nested containers (severe security risk). + +**Options**: +1. **External Kaniko service**: Deploy Kaniko or Buildkit as a separate service, agent submits build requests via API. +2. **Prohibit container builds**: Document that agents cannot build images, use static Dockerfile analysis only. +3. **Sidecar Kaniko pod** (Kubernetes only): Spawn ephemeral Kaniko sidecar for each agent run, agent communicates via shared volume. + +**Status**: No current agents require container builds. Defer until concrete use case emerges. If needed, external builder service is the architecturally sound option. + +### Multi-Architecture Support (arm64) + +**Problem**: The container image is currently built for linux/amd64 only. Some organizations use ARM-based runners (AWS Graviton, Apple Silicon for macOS GitHub Actions runners). + +**Options**: +1. **Multi-arch image**: Build and publish linux/amd64 and linux/arm64 variants. +2. **Architecture-specific images**: Separate image tags for each architecture. +3. **No ARM support**: Document amd64-only requirement. + +**Status**: OpenShell supports ARM64. Buildx can build multi-arch images. Requires testing on ARM runners and updating build pipeline. Defer until ARM runner adoption is significant. + +### Image Size Optimization + +**Problem**: Full runtime image is ~1.5-2GB. First pull on a new runner takes 30-60 seconds. + +**Options**: +1. **Multi-stage build optimization**: Remove build-time dependencies from final image. +2. **Distroless base**: Use distroless or minimal base image (Alpine) to reduce OS layer size. +3. **Layer caching**: Ensure runner Docker cache is persistent across jobs. +4. **Per-agent images**: Ship minimal base image, per-agent images add only required tools. + +**Status**: Current Dockerfile uses multi-stage builds. Distroless may conflict with OpenShell requirements (glibc, shell for scripts). Layer caching works on GitLab Docker executor but not GitHub Actions (ephemeral VMs). Per-agent images violate single-image architecture decision. Monitor image pull latency in production, optimize if it becomes a bottleneck. + +## References + +- [ADR-0029: Agent Execution Sandbox Architecture](../ADRs/0029-agent-execution-sandbox.md) +- [ADR-0017: Credential Isolation for Sandboxed Agents](../ADRs/0017-credential-isolation-for-sandboxed-agents.md) +- [ADR-0025: Provider Credential Delivery](../ADRs/0025-provider-credential-delivery-for-sandboxed-agents.md) +- [ADR-0028: GitLab Support Architecture](../ADRs/0028-gitlab-support.md) +- [agent-infrastructure.md](agent-infrastructure.md): Infrastructure layer exploration +- [OpenShell Documentation](https://docs.nvidia.com/openshell/) +- [Sigstore Cosign](https://docs.sigstore.dev/cosign/overview/) +- [GitLab Runner Documentation](https://docs.gitlab.com/runner/) From 66e7dd42ec1c804118a502e3a57fe5147ee4421d Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Tue, 5 May 2026 11:33:06 -0400 Subject: [PATCH 02/15] Address lint failures and review findings for ADR-0029 Lint fixes: - Remove gitlab-implementation from ADR frontmatter relates_to (doesn't exist yet) - Update references to ADR-0028 and gitlab-implementation.md to note they are pending in PR #601 Review findings (Medium): - Add builder stage to Dockerfile example (was missing, would fail to build) - Remove Docker socket mount from GitLab runner registration (security violation - bypasses sandbox isolation) - Update all references to pending GitLab support work (PR #601 instead of ADR-0028) Review findings (Low): - Remove placeholder OpenShell issue link, describe as upstream feature request instead - Replace non-existent docker/run@v1 action with standard docker run command - Add comment explaining docker login in GitLab job (for pulling additional images, not the base image) All changes address findings from fullsend-ai-review comment-only review. Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0029-agent-execution-sandbox.md | 8 ++--- docs/problems/agent-execution-environment.md | 33 ++++++++++++++------ 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index a109454169..6bf5167e29 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -4,7 +4,6 @@ status: Proposed relates_to: - agent-infrastructure - agent-execution-environment - - gitlab-implementation topics: - sandbox - container @@ -23,7 +22,7 @@ Proposed ## Context -Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support (ADR-0028) now under development, the execution architecture needs to work on both GitHub Actions and GitLab CI runners. +Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support now under development (proposed in PR #601), the execution architecture needs to work on both GitHub Actions and GitLab CI runners. The sandbox architecture has multiple concerns that need to be resolved together: @@ -33,7 +32,7 @@ The sandbox architecture has multiple concerns that need to be resolved together 4. **Resource limits**: CPU, memory, and timeout constraints need platform-independent expression 5. **OpenShell integration**: The sandbox runtime (OpenShell gateway + L7 policies + providers) must work in all executor environments -The [gitlab-implementation.md](../problems/gitlab-implementation.md) open questions section explicitly deferred this decision: "The agent execution environment is orthogonal to the CI/CD dispatch architecture. GitLab runner configuration, sandbox isolation, and compute architecture should be documented separately." +The GitLab support design (PR #601) explicitly deferred this decision: "The agent execution environment is orthogonal to the CI/CD dispatch architecture. GitLab runner configuration, sandbox isolation, and compute architecture should be documented separately." The forge abstraction (ADR-0005) keeps dispatch logic platform-neutral. This ADR addresses what happens *after* the dispatch pipeline triggers an agent job: how the agent actually executes. @@ -285,7 +284,6 @@ The implementation document is structured for iterative evolution as the sandbox - ADR-0005: Forge abstraction layer (dispatch is platform-neutral, execution must also be) - ADR-0017: Credential isolation for sandboxed agents (zero credentials in sandbox) - ADR-0025: Provider credential delivery (OpenShell providers for credential injection) -- ADR-0028: GitLab support architecture (dispatch pipelines, need execution layer) +- PR #601: GitLab support architecture (dispatch pipelines, explicitly deferred agent execution environment) - [agent-infrastructure.md](../problems/agent-infrastructure.md): Infrastructure layer exploration, SIG Agent Sandbox evaluation -- [gitlab-implementation.md](../problems/gitlab-implementation.md): Explicitly deferred agent execution environment - [OpenShell](https://github.com/NVIDIA/OpenShell): Sandbox runtime with L7 network policy enforcement diff --git a/docs/problems/agent-execution-environment.md b/docs/problems/agent-execution-environment.md index 2bd3ead68c..3d94296e87 100644 --- a/docs/problems/agent-execution-environment.md +++ b/docs/problems/agent-execution-environment.md @@ -25,6 +25,15 @@ The agent sandbox container image is the primary artifact that defines the execu The Dockerfile is organized in layers to optimize for cache reuse and minimize image size: ```dockerfile +# Builder stage: Compile fullsend CLI binary +FROM golang:1.23 AS builder +WORKDIR /app +COPY go.mod go.sum ./ +RUN go mod download +COPY cmd/ cmd/ +COPY internal/ internal/ +RUN CGO_ENABLED=0 GOOS=linux go build -o fullsend ./cmd/fullsend + # Base: Ubuntu 22.04 LTS (OpenShell requires glibc) FROM ubuntu:22.04 AS base @@ -273,10 +282,12 @@ jobs: timeout-minutes: 15 # Maximum 15 minutes per agent run steps: - name: Run agent - uses: docker/run@v1 - with: - image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 - options: --memory=4g --cpus=1.5 # Container resource limits + run: | + docker run --rm \ + --memory=4g --cpus=1.5 \ + -e AGENT_NAME=${{ inputs.agent_name }} \ + -e EVENT_PAYLOAD=${{ inputs.event_payload }} \ + ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 ``` ### GitLab CI (Docker Executor) @@ -383,7 +394,7 @@ Run the container as a non-root user inside a user namespace. The user appears a Replace iptables-based L7 enforcement with eBPF programs that hook into the network stack without requiring privileged containers. -**Status**: Not yet implemented in OpenShell. Upstream feature request: [NVIDIA/OpenShell#xyz](https://github.com/NVIDIA/OpenShell/issues/xyz) (placeholder, actual issue TBD). +**Status**: Not yet implemented in OpenShell. This would require an upstream feature request for eBPF-based L7 enforcement. #### Accept Privileged Requirement @@ -454,8 +465,9 @@ sudo gitlab-runner register \ --executor docker \ --description "fullsend-agent-runner" \ --docker-image "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" \ - --docker-privileged \ - --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" + --docker-privileged +# Note: --docker-privileged is required for OpenShell network namespace manipulation. +# Do NOT mount /var/run/docker.sock as this would bypass sandbox isolation. ``` **Configure resource limits** (`/etc/gitlab-runner/config.toml`): @@ -510,6 +522,9 @@ The agent sandbox image is public on ghcr.io, so no authentication is required f run-agent: image: registry.example.com/fullsend/agent-sandbox:v1.2.3 before_script: + # Registry authentication for pulling additional images during agent execution. + # The job's base image is already pulled by the runner before this script runs. + # This is only needed if agents pull additional images (e.g., for testing). - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY script: - fullsend run $AGENT_NAME @@ -631,9 +646,7 @@ The signature is stored in the container registry alongside the image. No privat ghcr.io/fullsend-ai/agent-sandbox:${{ inputs.image_tag }} - name: Run agent - uses: docker/run@v1 - with: - image: ghcr.io/fullsend-ai/agent-sandbox:${{ inputs.image_tag }} + run: docker run --rm ghcr.io/fullsend-ai/agent-sandbox:${{ inputs.image_tag }} ``` **GitLab CI:** From 5a4076b22fec4a2db3b2be0bfdb51840acc0f9f7 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Tue, 5 May 2026 11:44:06 -0400 Subject: [PATCH 03/15] Fix trailing whitespace in agent-execution-environment.md Co-Authored-By: Claude Sonnet 4.5 --- docs/problems/agent-execution-environment.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/problems/agent-execution-environment.md b/docs/problems/agent-execution-environment.md index 3d94296e87..1a49ea80db 100644 --- a/docs/problems/agent-execution-environment.md +++ b/docs/problems/agent-execution-environment.md @@ -188,7 +188,7 @@ OpenShell gateway configuration is embedded in the container image at `/etc/open # OpenShell gateway configuration api: listen: 127.0.0.1:8080 # Gateway API (sandbox creation, policy management) - + proxy: listen: 0.0.0.0:3128 # HTTP proxy for sandbox egress (L7 policy enforcement) @@ -234,12 +234,12 @@ rules: - endpoint: "https://api.github.com/repos/*/*/issues/*" methods: [GET] binaries: [gh, curl] # Only gh and curl can call this endpoint - + # Allow listing issues - endpoint: "https://api.github.com/repos/*/*/issues" methods: [GET] binaries: [gh, curl] - + # Deny all other GitHub API calls - endpoint: "https://api.github.com/**" methods: [GET, POST, PUT, PATCH, DELETE] @@ -299,7 +299,7 @@ GitLab runner with Docker executor supports container resource limits via runner [[runners]] name = "fullsend-agent-runner" executor = "docker" - + [runners.docker] image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" privileged = true # Required for OpenShell network namespace manipulation @@ -327,17 +327,17 @@ Kubernetes executor uses pod resource requests and limits: [[runners]] name = "fullsend-k8s-runner" executor = "kubernetes" - + [runners.kubernetes] image = "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" namespace = "fullsend-agents" privileged = true - + cpu_request = "1" cpu_limit = "1.5" memory_request = "2Gi" memory_limit = "4Gi" - + service_cpu_request = "0.1" service_memory_request = "128Mi" ``` From 866d8c2fd3fda7417894aa130b57c53cfbed3940 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Tue, 5 May 2026 12:48:19 -0400 Subject: [PATCH 04/15] Fix broken link to ADR-0028 (pending in PR #601) Replace markdown link to non-existent ADR-0028 file with plain text noting it is pending in PR #601. This prevents broken link if this PR merges before #601. Addresses review finding about broken link in References section. Co-Authored-By: Claude Sonnet 4.5 --- docs/problems/agent-execution-environment.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/problems/agent-execution-environment.md b/docs/problems/agent-execution-environment.md index 1a49ea80db..22935d4dd4 100644 --- a/docs/problems/agent-execution-environment.md +++ b/docs/problems/agent-execution-environment.md @@ -913,7 +913,7 @@ spec: - [ADR-0029: Agent Execution Sandbox Architecture](../ADRs/0029-agent-execution-sandbox.md) - [ADR-0017: Credential Isolation for Sandboxed Agents](../ADRs/0017-credential-isolation-for-sandboxed-agents.md) - [ADR-0025: Provider Credential Delivery](../ADRs/0025-provider-credential-delivery-for-sandboxed-agents.md) -- [ADR-0028: GitLab Support Architecture](../ADRs/0028-gitlab-support.md) +- ADR-0028: GitLab Support Architecture (pending in PR #601) - [agent-infrastructure.md](agent-infrastructure.md): Infrastructure layer exploration - [OpenShell Documentation](https://docs.nvidia.com/openshell/) - [Sigstore Cosign](https://docs.sigstore.dev/cosign/overview/) From 73604fd8f0fb4711a4a1befbc4445f7cbad89074 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 15:46:04 -0400 Subject: [PATCH 05/15] Address review feedback on ADR-0029 and implementation doc **From @maruiz93:** - Add note about Fedora/Podman compatibility consideration in image composition - Clarify language runtimes are for fullsend's built-in agents; BYOA can customize - Clarify harness role as control plane for agent execution - Add disadvantage note about OpenShell feature parity validation needed for Kubernetes vs Docker **From fullsend-ai-review bot:** - Add note that OpenShell configuration examples are illustrative and subject to validation - Enhance implementation doc header to clarify it's an ADR companion, not a problem exploration Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0029-agent-execution-sandbox.md | 9 ++++++--- docs/problems/agent-execution-environment.md | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index 6bf5167e29..7c377d015b 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -79,10 +79,10 @@ run-agent: - GitLab shell executor cannot use this (no container runtime) **Image composition strategy:** -- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu) -- Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend agents require, not arbitrary user code +- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu). Note: Fedora-based images may be considered when exploring rootless Podman support, though `fullsend run` would need compatibility testing for Fedora/RHEL environments. +- Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend's built-in agents require, not arbitrary user code. Organizations implementing "Bring Your Own Agent" can build customized images with different runtime sets (see "Image Build and Distribution" in Open Questions). - Layer 2: Common tools (git, gh CLI, curl, jq) -- Layer 3: Agent harness (`fullsend run` CLI) +- Layer 3: Agent harness (`fullsend run` CLI) — provides the control plane for agent execution, sandbox initialization, and policy enforcement - Layer 4: Provider definitions and policy templates **OpenShell integration:** @@ -144,6 +144,7 @@ Similar `kubectl` calls from CI jobs. - Agent runs are asynchronous API calls (harder to stream logs to CI job output) - Kubernetes SIG Agent Sandbox evaluation (agent-infrastructure.md) notes poor fit for ephemeral task-scoped execution - Adds latency (API call + pod scheduling + image pull vs direct container start in CI executor) +- OpenShell feature parity between Kubernetes pod networking and standard Docker networking needs validation — pod network namespaces, CNI plugins, and service mesh sidecars may interact differently with OpenShell's L7 proxy than direct container networking ### Option 4: Minimal Sandbox + Dynamic Tool Installation @@ -210,6 +211,8 @@ Detailed implementation guidance has been moved to [docs/problems/agent-executio - Image signing and verification (Sigstore, cosign) - Upgrade and rollback procedures +**Note:** OpenShell configuration examples (version numbers, API endpoints, configuration syntax) are illustrative and based on design-phase exploration. These details should be validated against the actual OpenShell release used during implementation, as APIs and configuration formats may evolve. + The implementation document is structured for iterative evolution as the sandbox architecture is validated in production. ## Consequences diff --git a/docs/problems/agent-execution-environment.md b/docs/problems/agent-execution-environment.md index 22935d4dd4..f5adffe346 100644 --- a/docs/problems/agent-execution-environment.md +++ b/docs/problems/agent-execution-environment.md @@ -2,7 +2,7 @@ How do fullsend agents execute on CI runners, what does the sandbox environment contain, and how does it work across GitHub Actions and GitLab CI? -This document contains implementation details for the agent execution sandbox architecture. For the architectural decision and rationale, see [ADR-0029](../ADRs/0029-agent-execution-sandbox.md). +**Note:** This is an implementation companion to [ADR-0029](../ADRs/0029-agent-execution-sandbox.md), not a problem exploration. It provides detailed implementation guidance for the chosen sandbox architecture, structured for iterative evolution as the design is validated in production. Once the architecture stabilizes and moves from "Proposed" to "Accepted", operational content may migrate to `docs/guides/` per ADR-0023. ## Table of Contents From 6c94c8ac4d7ee34949dc06bfa57e8c9bb9b57a2f Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 15:47:00 -0400 Subject: [PATCH 06/15] Move agent-execution-environment.md from problems to plans This implementation document is a companion to ADR-0029, not a problem exploration. Moving it to docs/plans/ better reflects its purpose as implementation guidance. Co-Authored-By: Claude Sonnet 4.5 --- README.md | 2 +- docs/ADRs/0029-agent-execution-sandbox.md | 2 +- docs/{problems => plans}/agent-execution-environment.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) rename docs/{problems => plans}/agent-execution-environment.md (98%) diff --git a/README.md b/README.md index e2f848efc5..fdf67917f0 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ This is not a product spec. It's an evolving exploration of a hard problem space - [Performance Verification](docs/problems/performance-verification.md) — Catching agent-introduced performance regressions before they reach production - [Production Feedback](docs/problems/production-feedback.md) — How platform execution signals feed back into what agents work on and how they assess risk - [Testing the Agents](docs/problems/testing-agents.md) — CI for prompts: regression testing, eval frameworks, and behavioral verification for agent instructions - - [Agent Execution Environment](docs/problems/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI + - [Agent Execution Environment](docs/plans/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI - [GitLab Implementation](docs/problems/gitlab-implementation.md) — Implementation details for GitLab support: webhook security, dispatch pipelines, forge interface evolution - [Operational Observability](docs/problems/operational-observability.md) — How do the humans operating an autonomous software factory understand what it is doing, debug it when it goes wrong, and improve it over time? - [Platform Nativeness](docs/problems/platform-nativeness.md) — When the platform you automate is also the one you build on: which problems are inherent vs. self-inflicted diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index 7c377d015b..73dce579dd 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -200,7 +200,7 @@ Dynamic tool installation violates the zero-trust execution principle: agents sh ## Implementation Details -Detailed implementation guidance has been moved to [docs/problems/agent-execution-environment.md](../problems/agent-execution-environment.md), including: +Detailed implementation guidance has been moved to [docs/plans/agent-execution-environment.md](../plans/agent-execution-environment.md), including: - Container image build pipeline and versioning strategy - OpenShell configuration for nested sandbox creation diff --git a/docs/problems/agent-execution-environment.md b/docs/plans/agent-execution-environment.md similarity index 98% rename from docs/problems/agent-execution-environment.md rename to docs/plans/agent-execution-environment.md index f5adffe346..b3cae0ce2e 100644 --- a/docs/problems/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -2,7 +2,7 @@ How do fullsend agents execute on CI runners, what does the sandbox environment contain, and how does it work across GitHub Actions and GitLab CI? -**Note:** This is an implementation companion to [ADR-0029](../ADRs/0029-agent-execution-sandbox.md), not a problem exploration. It provides detailed implementation guidance for the chosen sandbox architecture, structured for iterative evolution as the design is validated in production. Once the architecture stabilizes and moves from "Proposed" to "Accepted", operational content may migrate to `docs/guides/` per ADR-0023. +**Note:** This is an implementation plan companion to [ADR-0029](../ADRs/0029-agent-execution-sandbox.md). It provides detailed implementation guidance for the chosen sandbox architecture, structured for iterative evolution as the design is validated in production. Once the architecture stabilizes and moves from "Proposed" to "Accepted", operational content may migrate to `docs/guides/` per ADR-0023. ## Table of Contents From dc599755b19a8cc9d8a42709aa7adfa4fc40a115 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 17:31:28 -0400 Subject: [PATCH 07/15] Fix medium and low review findings from latest bot review **Medium:** - Add docs/plans/ section to README for proper categorization - Update stale ADR-0028 reference in implementation doc (was "pending in PR #601") **Low:** - Update ADR context to reflect GitLab support is decided (ADR-0028), not "under development" - Update all references from "PR #601" to ADR-0028 throughout **Other:** - Remove agent-execution-environment from ADR frontmatter relates_to (now in plans, not problems) - Fix agent-infrastructure.md link path in implementation doc references Co-Authored-By: Claude Sonnet 4.5 --- README.md | 3 ++- docs/ADRs/0029-agent-execution-sandbox.md | 7 +++---- docs/plans/agent-execution-environment.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index fdf67917f0..7bb7edb0aa 100644 --- a/README.md +++ b/README.md @@ -33,13 +33,14 @@ This is not a product spec. It's an evolving exploration of a hard problem space - [Performance Verification](docs/problems/performance-verification.md) — Catching agent-introduced performance regressions before they reach production - [Production Feedback](docs/problems/production-feedback.md) — How platform execution signals feed back into what agents work on and how they assess risk - [Testing the Agents](docs/problems/testing-agents.md) — CI for prompts: regression testing, eval frameworks, and behavioral verification for agent instructions - - [Agent Execution Environment](docs/plans/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI - [GitLab Implementation](docs/problems/gitlab-implementation.md) — Implementation details for GitLab support: webhook security, dispatch pipelines, forge interface evolution - [Operational Observability](docs/problems/operational-observability.md) — How do the humans operating an autonomous software factory understand what it is doing, debug it when it goes wrong, and improve it over time? - [Platform Nativeness](docs/problems/platform-nativeness.md) — When the platform you automate is also the one you build on: which problems are inherent vs. self-inflicted - [Cross-Run Memory](docs/problems/cross-run-memory.md) — How agents learn from prior run outcomes without violating the ephemeral sandbox invariant - **[docs/problems/applied/](docs/problems/applied/)** — Organization-specific considerations for downstream consumers: - [konflux-ci](docs/problems/applied/konflux-ci/) — Kubernetes-native CI/CD platform (the original proving ground) +- **[docs/plans/](docs/plans/)** — Implementation plans for accepted or in-progress designs: + - [Agent Execution Environment](docs/plans/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI - **[docs/guides/](docs/guides/)** — Practical how-to documentation for administrators and developers (see [ADR 0023](docs/ADRs/0023-user-documentation-structure.md)) - **[docs/ADRs/](docs/ADRs/)** — Architecture Decision Records for crystallizing specific decisions (see [ADR 0001](docs/ADRs/0001-use-adrs-for-decision-making.md)) - **[web/](web/)** — Browser-delivered assets for the public site (document graph today; future Vite app here). Cloudflare Worker config lives in [`cloudflare_site/`](cloudflare_site/) ([ADR 0019](docs/ADRs/0019-web-source-and-cloudflare-site-layout.md)). diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index 73dce579dd..b08c4225c8 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -3,7 +3,6 @@ title: "29. Agent Execution Sandbox Architecture" status: Proposed relates_to: - agent-infrastructure - - agent-execution-environment topics: - sandbox - container @@ -22,7 +21,7 @@ Proposed ## Context -Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support now under development (proposed in PR #601), the execution architecture needs to work on both GitHub Actions and GitLab CI runners. +Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support decided (ADR-0028), the execution architecture needs to work on both GitHub Actions and GitLab CI runners. The sandbox architecture has multiple concerns that need to be resolved together: @@ -32,7 +31,7 @@ The sandbox architecture has multiple concerns that need to be resolved together 4. **Resource limits**: CPU, memory, and timeout constraints need platform-independent expression 5. **OpenShell integration**: The sandbox runtime (OpenShell gateway + L7 policies + providers) must work in all executor environments -The GitLab support design (PR #601) explicitly deferred this decision: "The agent execution environment is orthogonal to the CI/CD dispatch architecture. GitLab runner configuration, sandbox isolation, and compute architecture should be documented separately." +The GitLab support design (ADR-0028) explicitly deferred this decision: "The agent execution environment is orthogonal to the CI/CD dispatch architecture. GitLab runner configuration, sandbox isolation, and compute architecture should be documented separately." The forge abstraction (ADR-0005) keeps dispatch logic platform-neutral. This ADR addresses what happens *after* the dispatch pipeline triggers an agent job: how the agent actually executes. @@ -287,6 +286,6 @@ The implementation document is structured for iterative evolution as the sandbox - ADR-0005: Forge abstraction layer (dispatch is platform-neutral, execution must also be) - ADR-0017: Credential isolation for sandboxed agents (zero credentials in sandbox) - ADR-0025: Provider credential delivery (OpenShell providers for credential injection) -- PR #601: GitLab support architecture (dispatch pipelines, explicitly deferred agent execution environment) +- [ADR-0028: GitLab Support Architecture](0028-gitlab-support.md) (dispatch pipelines, explicitly deferred agent execution environment) - [agent-infrastructure.md](../problems/agent-infrastructure.md): Infrastructure layer exploration, SIG Agent Sandbox evaluation - [OpenShell](https://github.com/NVIDIA/OpenShell): Sandbox runtime with L7 network policy enforcement diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index b3cae0ce2e..91ceb3b128 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -913,8 +913,8 @@ spec: - [ADR-0029: Agent Execution Sandbox Architecture](../ADRs/0029-agent-execution-sandbox.md) - [ADR-0017: Credential Isolation for Sandboxed Agents](../ADRs/0017-credential-isolation-for-sandboxed-agents.md) - [ADR-0025: Provider Credential Delivery](../ADRs/0025-provider-credential-delivery-for-sandboxed-agents.md) -- ADR-0028: GitLab Support Architecture (pending in PR #601) -- [agent-infrastructure.md](agent-infrastructure.md): Infrastructure layer exploration +- [ADR-0028: GitLab Support Architecture](../ADRs/0028-gitlab-support.md) +- [agent-infrastructure.md](../problems/agent-infrastructure.md): Infrastructure layer exploration - [OpenShell Documentation](https://docs.nvidia.com/openshell/) - [Sigstore Cosign](https://docs.sigstore.dev/cosign/overview/) - [GitLab Runner Documentation](https://docs.gitlab.com/runner/) From 27409bf4dba35b8d79f048bd5e9afcbd74fe3ccc Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 17:42:51 -0400 Subject: [PATCH 08/15] Fix low severity review findings **Low:** - Complete README docs/plans/ index with all three existing plan files - Update GitLab runner registration to use modern authentication tokens (glrt-) instead of deprecated registration tokens, with note about GitLab 15.10+ requirement Co-Authored-By: Claude Sonnet 4.5 --- README.md | 2 ++ docs/plans/agent-execution-environment.md | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7bb7edb0aa..3d19b5af22 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,9 @@ This is not a product spec. It's an evolving exploration of a hard problem space - **[docs/problems/applied/](docs/problems/applied/)** — Organization-specific considerations for downstream consumers: - [konflux-ci](docs/problems/applied/konflux-ci/) — Kubernetes-native CI/CD platform (the original proving ground) - **[docs/plans/](docs/plans/)** — Implementation plans for accepted or in-progress designs: + - [ADR-0046 Drift Scanner](docs/plans/2026-03-06-adr46-drift-scanner.md) — Implementation plan for building a drift scanner to detect Tekton tasks using non-compliant images - [Agent Execution Environment](docs/plans/agent-execution-environment.md) — Container image design, OpenShell sandbox configuration, resource limits, and cross-platform execution on GitHub Actions and GitLab CI + - [Vertex AI Inference Provisioning](docs/plans/vertex-inference-provisioning.md) — Credential provisioning and configuration for GCP Vertex AI inference provider - **[docs/guides/](docs/guides/)** — Practical how-to documentation for administrators and developers (see [ADR 0023](docs/ADRs/0023-user-documentation-structure.md)) - **[docs/ADRs/](docs/ADRs/)** — Architecture Decision Records for crystallizing specific decisions (see [ADR 0001](docs/ADRs/0001-use-adrs-for-decision-making.md)) - **[web/](web/)** — Browser-delivered assets for the public site (document graph today; future Vite app here). Cloudflare Worker config lives in [`cloudflare_site/`](cloudflare_site/) ([ADR 0019](docs/ADRs/0019-web-source-and-cloudflare-site-layout.md)). diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index 91ceb3b128..dd788366db 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -458,14 +458,28 @@ sudo apt-get install gitlab-runner ``` **Register runner:** + +> **Note:** GitLab deprecated registration tokens in favor of runner authentication tokens (starting with `glrt-`) in GitLab 15.10. The `--registration-token` flag was removed in GitLab 17.0. For GitLab 15.10+, create a runner authentication token in the GitLab UI (Settings → CI/CD → Runners → New runner) and use `--token` instead. + ```bash +# For GitLab 15.10+ (recommended): sudo gitlab-runner register \ --url https://gitlab.com \ - --registration-token $REGISTRATION_TOKEN \ + --token $RUNNER_AUTHENTICATION_TOKEN \ --executor docker \ --description "fullsend-agent-runner" \ --docker-image "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" \ --docker-privileged + +# For GitLab < 15.10 (legacy): +# sudo gitlab-runner register \ +# --url https://gitlab.com \ +# --registration-token $REGISTRATION_TOKEN \ +# --executor docker \ +# --description "fullsend-agent-runner" \ +# --docker-image "ghcr.io/fullsend-ai/agent-sandbox:v1.2.3" \ +# --docker-privileged + # Note: --docker-privileged is required for OpenShell network namespace manipulation. # Do NOT mount /var/run/docker.sock as this would bypass sandbox isolation. ``` From 0c06fd7cfa143b2395c5b4e0a22cd708118d09ad Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 17:54:18 -0400 Subject: [PATCH 09/15] Fix remaining medium and low review findings for approval **Medium:** - Update Kubernetes Helm install to use modern runnerToken instead of deprecated runnerRegistrationToken - Fix PodSecurityPolicy deprecation timeline (deprecated in 1.21, removed in 1.25, not 1.29) **Low:** - Move Fedora/Podman note from image composition to Rootless OpenShell Support section for better flow Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0029-agent-execution-sandbox.md | 4 ++-- docs/plans/agent-execution-environment.md | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index b08c4225c8..cd662be937 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -78,7 +78,7 @@ run-agent: - GitLab shell executor cannot use this (no container runtime) **Image composition strategy:** -- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu). Note: Fedora-based images may be considered when exploring rootless Podman support, though `fullsend run` would need compatibility testing for Fedora/RHEL environments. +- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu) - Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend's built-in agents require, not arbitrary user code. Organizations implementing "Bring Your Own Agent" can build customized images with different runtime sets (see "Image Build and Distribution" in Open Questions). - Layer 2: Common tools (git, gh CLI, curl, jq) - Layer 3: Agent harness (`fullsend run` CLI) — provides the control plane for agent execution, sandbox initialization, and policy enforcement @@ -257,7 +257,7 @@ The implementation document is structured for iterative evolution as the sandbox 2. **OpenShell rootless mode**: Upstream feature request to support L7 policy enforcement without privileged containers (e.g., via eBPF or SECCOMP). 3. **Platform exemption**: Document that fullsend requires privileged containers and provide guidance for organizations to create PodSecurityPolicy exemptions for agent workloads. -**Status**: User namespace remapping is the most viable near-term path. Requires testing on GitHub Actions (Docker-in-VM) and GitLab Kubernetes executor (pod security contexts). OpenShell rootless mode is ideal long-term but depends on upstream. +**Status**: User namespace remapping is the most viable near-term path. Requires testing on GitHub Actions (Docker-in-VM) and GitLab Kubernetes executor (pod security contexts). OpenShell rootless mode is ideal long-term but depends on upstream. Note: Fedora-based base images may be considered when exploring rootless Podman support, though `fullsend run` would need compatibility testing for Fedora/RHEL environments. ### Image Build and Distribution diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index dd788366db..7a6f82620d 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -411,7 +411,7 @@ Document that fullsend agents require privileged containers and provide guidance For organizations using Kubernetes with PodSecurityPolicies (or Pod Security Standards in Kubernetes 1.25+): ```yaml -# PodSecurityPolicy (deprecated in K8s 1.25, removed in 1.29) +# PodSecurityPolicy (deprecated in K8s 1.21, removed in 1.25) apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: @@ -502,13 +502,25 @@ sudo gitlab-runner register \ ### Kubernetes Executor Setup **Install GitLab Runner as a Kubernetes deployment:** + +> **Note:** For GitLab 15.10+, create a runner authentication token in the GitLab UI (Settings → CI/CD → Runners → New runner) and use `runnerToken` instead of the deprecated `runnerRegistrationToken` (removed in GitLab 17.0). + ```bash helm repo add gitlab https://charts.gitlab.io + +# For GitLab 15.10+ (recommended): helm install gitlab-runner gitlab/gitlab-runner \ --namespace fullsend-agents \ - --set runnerRegistrationToken=$REGISTRATION_TOKEN \ + --set runnerToken=$RUNNER_AUTHENTICATION_TOKEN \ --set rbac.create=true \ --set runners.privileged=true + +# For GitLab < 15.10 (legacy): +# helm install gitlab-runner gitlab/gitlab-runner \ +# --namespace fullsend-agents \ +# --set runnerRegistrationToken=$REGISTRATION_TOKEN \ +# --set rbac.create=true \ +# --set runners.privileged=true ``` **Configure executor** (values.yaml): From a53603ddd391a7d50e1aef214a3b8864466cb5cb Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 18:06:36 -0400 Subject: [PATCH 10/15] Address ADR-0030 alignment issues and golang package name **Medium:** - Add prominent warning note that OpenShell details are illustrative and must follow ADR-0030's decisions - Fix Dockerfile ENTRYPOINT to use 'openshell gateway start' per ADR-0030 - Clarify that ADR-0030 decides CLI-based interaction (not REST API), SSH execution, SCP file delivery, and provider-based credentials **Low:** - Fix golang installation in Dockerfile - copy from official image instead of using invalid package name The implementation plan retains illustrative examples for reference but clearly notes they must be validated against ADR-0030 during implementation. Co-Authored-By: Claude Sonnet 4.5 --- docs/plans/agent-execution-environment.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index 7a6f82620d..cb1f9e5cb2 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -52,13 +52,17 @@ RUN curl -L https://github.com/NVIDIA/OpenShell/releases/download/${OPENSHELL_VE # Language runtimes layer (can be cached independently) FROM base AS runtimes +# Note: golang-1.23 is not available in default Ubuntu 22.04 repos. +# Use official golang image for builder stage (as shown above) or add PPA/manual download for runtime. +# Example: COPY --from=golang:1.23 /usr/local/go /usr/local/go RUN apt-get update && apt-get install -y \ - golang-1.23 \ python3.11 \ python3-pip \ nodejs \ npm \ && rm -rf /var/lib/apt/lists/* +COPY --from=golang:1.23 /usr/local/go /usr/local/go +ENV PATH="/usr/local/go/bin:${PATH}" # Tools layer FROM runtimes AS tools @@ -84,8 +88,8 @@ FROM harness AS final COPY policies/ /opt/fullsend/policies/ COPY providers/ /opt/fullsend/providers/ -# OpenShell gateway runs as PID 1 -ENTRYPOINT ["/usr/local/bin/openshell", "gateway"] +# OpenShell gateway runs as PID 1 (per ADR-0030) +ENTRYPOINT ["/usr/local/bin/openshell", "gateway", "start"] ``` ### Build Pipeline (GitHub Actions) @@ -178,7 +182,9 @@ Config repo templates reference explicit semver tags: `ghcr.io/fullsend-ai/agent ## OpenShell Configuration -OpenShell runs as the container entrypoint (PID 1). When `fullsend run` is invoked, it communicates with the OpenShell gateway API to create nested sandboxes for individual agent processes. +> **⚠️ Important:** The OpenShell interaction details in this section are illustrative and based on design-phase exploration. **[ADR-0030](../ADRs/0030-openshell-sandbox-interaction-model.md) (Accepted)** decides the actual interaction model: CLI-based sandbox creation (`openshell sandbox create`), SSH for command execution via HTTP CONNECT tunnels, SCP for file delivery during bootstrap, and provider-based credential delivery via `openshell provider create`. Files like agent definitions and skills are SCP'd during bootstrap, not baked into the image. **Implementation must follow ADR-0030's decisions.** The API endpoints, configuration file formats, and workflow described below may not match the actual OpenShell CLI/API. + +OpenShell runs as the container entrypoint (PID 1). When `fullsend run` is invoked, it communicates with the OpenShell gateway to create nested sandboxes for individual agent processes. ### Gateway Configuration From 665de4b43d8b7f91e7930e85141a3489a284b60f Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 18:18:02 -0400 Subject: [PATCH 11/15] Fix final medium and low review findings for clean approval **Medium:** - Remove deprecated PodSecurityPolicy YAML example entirely (API removed in K8s 1.25) - Keep only Pod Security Standards (modern, recommended approach) **Low:** - Replace brittle 'sleep 5' with proper readiness check in rootless Docker example - Fix ADR-0028 status reference from "decided" to "proposed" (matches actual status) All findings now addressed - only Info items remain. Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0029-agent-execution-sandbox.md | 2 +- docs/plans/agent-execution-environment.md | 27 +++-------------------- 2 files changed, 4 insertions(+), 25 deletions(-) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index cd662be937..b9e94171aa 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -21,7 +21,7 @@ Proposed ## Context -Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support decided (ADR-0028), the execution architecture needs to work on both GitHub Actions and GitLab CI runners. +Fullsend agents execute within isolated sandboxes that enforce security boundaries: filesystem access control, network policy enforcement, and credential isolation (ADR-0017, ADR-0025). The current implementation uses OpenShell with per-agent L7 network policies and runs on GitHub Actions runners. With GitLab support proposed (ADR-0028), the execution architecture needs to work on both GitHub Actions and GitLab CI runners. The sandbox architecture has multiple concerns that need to be resolved together: diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index cb1f9e5cb2..788f609b4f 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -378,7 +378,7 @@ Run the container as a non-root user inside a user namespace. The user appears a - name: Run agent with rootless Docker run: | dockerd-rootless.sh & - sleep 5 + until docker info >/dev/null 2>&1; do sleep 1; done docker run --rm \ -e AGENT_NAME=triage \ ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 @@ -414,30 +414,9 @@ Document that fullsend agents require privileged containers and provide guidance ### Kubernetes PodSecurityPolicy Configuration -For organizations using Kubernetes with PodSecurityPolicies (or Pod Security Standards in Kubernetes 1.25+): +For organizations using Kubernetes, configure pod security via Pod Security Standards (Kubernetes 1.25+): -```yaml -# PodSecurityPolicy (deprecated in K8s 1.21, removed in 1.25) -apiVersion: policy/v1beta1 -kind: PodSecurityPolicy -metadata: - name: fullsend-agent-privileged -spec: - privileged: true - allowPrivilegeEscalation: true - allowedCapabilities: - - CAP_NET_ADMIN - fsGroup: - rule: RunAsAny - runAsUser: - rule: RunAsAny - seLinux: - rule: RunAsAny - volumes: - - '*' -``` - -**Pod Security Standards (K8s 1.25+):** +**Pod Security Standards (recommended - K8s 1.25+):** ```yaml apiVersion: v1 kind: Namespace From 73085820fc7c2dee6b5da0b9c382c5653a13f954 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Sat, 9 May 2026 18:29:14 -0400 Subject: [PATCH 12/15] Final cleanup: fix sandbox confusion and layer description **Medium:** - Remove confusing docker login example from GitLab CI config - Clarify that base image pull is handled by runner, not by agents in sandbox - Agents do NOT have Docker daemon access (isolated by OpenShell) **Low:** - Remove parenthetical explanation from Layer 3 for consistency with other layers Iteration 7/7 complete. Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0029-agent-execution-sandbox.md | 2 +- docs/plans/agent-execution-environment.md | 8 +++----- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0029-agent-execution-sandbox.md index b9e94171aa..0c7b41e515 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0029-agent-execution-sandbox.md @@ -81,7 +81,7 @@ run-agent: - Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu) - Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend's built-in agents require, not arbitrary user code. Organizations implementing "Bring Your Own Agent" can build customized images with different runtime sets (see "Image Build and Distribution" in Open Questions). - Layer 2: Common tools (git, gh CLI, curl, jq) -- Layer 3: Agent harness (`fullsend run` CLI) — provides the control plane for agent execution, sandbox initialization, and policy enforcement +- Layer 3: Agent harness (`fullsend run` CLI) - Layer 4: Provider definitions and policy templates **OpenShell integration:** diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index 788f609b4f..e10c1c0412 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -532,11 +532,9 @@ The agent sandbox image is public on ghcr.io, so no authentication is required f # .gitlab-ci.yml run-agent: image: registry.example.com/fullsend/agent-sandbox:v1.2.3 - before_script: - # Registry authentication for pulling additional images during agent execution. - # The job's base image is already pulled by the runner before this script runs. - # This is only needed if agents pull additional images (e.g., for testing). - - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY + # Note: The job's base image is pulled by the runner using its configured credentials + # (CI_REGISTRY_USER/PASSWORD, image_pull_secrets, or runner config). Agents inside + # the sandbox do NOT have Docker daemon access - they are isolated by OpenShell. script: - fullsend run $AGENT_NAME ``` From a6b6eb00ba5d00e86af875976488c4b6d60e39a7 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Mon, 11 May 2026 10:06:14 -0400 Subject: [PATCH 13/15] Renumber ADR from 0029 to 0036 Update agent execution sandbox ADR number from 0029 to 0036 to avoid conflicts with ADRs merged to main. Update all references in agent-execution-environment.md. Co-Authored-By: Claude Sonnet 4.5 --- ...t-execution-sandbox.md => 0036-agent-execution-sandbox.md} | 4 ++-- docs/plans/agent-execution-environment.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) rename docs/ADRs/{0029-agent-execution-sandbox.md => 0036-agent-execution-sandbox.md} (99%) diff --git a/docs/ADRs/0029-agent-execution-sandbox.md b/docs/ADRs/0036-agent-execution-sandbox.md similarity index 99% rename from docs/ADRs/0029-agent-execution-sandbox.md rename to docs/ADRs/0036-agent-execution-sandbox.md index 0c7b41e515..76c4d323a7 100644 --- a/docs/ADRs/0029-agent-execution-sandbox.md +++ b/docs/ADRs/0036-agent-execution-sandbox.md @@ -1,5 +1,5 @@ --- -title: "29. Agent Execution Sandbox Architecture" +title: "36. Agent Execution Sandbox Architecture" status: Proposed relates_to: - agent-infrastructure @@ -11,7 +11,7 @@ topics: - openshell --- -# 29. Agent Execution Sandbox Architecture +# 36. Agent Execution Sandbox Architecture Date: 2026-05-05 diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index e10c1c0412..4b61a7dcec 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -2,7 +2,7 @@ How do fullsend agents execute on CI runners, what does the sandbox environment contain, and how does it work across GitHub Actions and GitLab CI? -**Note:** This is an implementation plan companion to [ADR-0029](../ADRs/0029-agent-execution-sandbox.md). It provides detailed implementation guidance for the chosen sandbox architecture, structured for iterative evolution as the design is validated in production. Once the architecture stabilizes and moves from "Proposed" to "Accepted", operational content may migrate to `docs/guides/` per ADR-0023. +**Note:** This is an implementation plan companion to [ADR-0036](../ADRs/0036-agent-execution-sandbox.md). It provides detailed implementation guidance for the chosen sandbox architecture, structured for iterative evolution as the design is validated in production. Once the architecture stabilizes and moves from "Proposed" to "Accepted", operational content may migrate to `docs/guides/` per ADR-0023. ## Table of Contents From cb8a17fb54c78d64a7f541d6676b4146b504f3aa Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Mon, 11 May 2026 10:13:11 -0400 Subject: [PATCH 14/15] Address PR review feedback on ADR-0036 - Mention Fedora/RHEL as base image alternatives for Podman deployments - Add explicit note about BYOA image customization in Decision section - Clarify GitLab CI script: field usage vs GitHub Actions - OpenShell K8s/Docker parity already addressed in Option 3 disadvantages Addresses review comments from maruiz93 and ifireball. Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0036-agent-execution-sandbox.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/ADRs/0036-agent-execution-sandbox.md b/docs/ADRs/0036-agent-execution-sandbox.md index 76c4d323a7..de6ca33a17 100644 --- a/docs/ADRs/0036-agent-execution-sandbox.md +++ b/docs/ADRs/0036-agent-execution-sandbox.md @@ -58,7 +58,7 @@ Package the entire agent execution environment (OpenShell, agent harness, tools, run-agent: image: ghcr.io/fullsend-ai/agent-sandbox:v1.2.3 script: - - fullsend run $AGENT_NAME + - fullsend run $AGENT_NAME # GitLab CI uses script: field for commands to run inside the container ``` **Isolation mechanism:** Container runtime (Docker, Podman, GitLab's Docker executor). OpenShell creates nested sandboxes inside the container for individual agent processes. @@ -78,7 +78,7 @@ run-agent: - GitLab shell executor cannot use this (no container runtime) **Image composition strategy:** -- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu) +- Base: OpenShell-enabled minimal Linux (Alpine or Ubuntu for Docker; Fedora/RHEL for Podman-based deployments) - Layer 1: Language runtimes (Go, Python, Node.js) — only what fullsend's built-in agents require, not arbitrary user code. Organizations implementing "Bring Your Own Agent" can build customized images with different runtime sets (see "Image Build and Distribution" in Open Questions). - Layer 2: Common tools (git, gh CLI, curl, jq) - Layer 3: Agent harness (`fullsend run` CLI) @@ -181,13 +181,15 @@ A single container image (`ghcr.io/fullsend-ai/agent-sandbox`) contains OpenShel - GitHub Actions Windows/macOS runners (OpenShell Linux-only, not a current fullsend target) **Image composition:** -- Base: Ubuntu 22.04 (OpenShell requires glibc, Alpine musl incompatible) +- Base: Ubuntu 22.04 (OpenShell requires glibc, Alpine musl incompatible; Fedora/RHEL alternative for Podman-first environments) - OpenShell 0.0.37-dev+ with Podman support -- Language runtimes: Go 1.23, Python 3.11, Node.js 20 (LTS) +- Language runtimes: Go 1.23, Python 3.11, Node.js 20 (LTS) — built-in agent requirements only - Tools: git, gh CLI, curl, jq, yq - Agent harness: `fullsend run` CLI binary - Provider templates and L7 policy examples in `/opt/fullsend/policies/` +**Note on "Bring Your Own Agent":** The reference image contains language runtimes for fullsend's built-in agents (triage, code, review, fix). Organizations implementing custom agents with different runtime requirements can build customized images from the base Dockerfile, replacing or extending the language runtime layer. See "Image Build and Distribution" in Open Questions for per-org image build strategies. + **Why Docker-first over platform-specific (Option 2):** Maintaining environment parity across GitHub and GitLab is a security requirement, not just operational convenience. Security boundaries (L7 policies, credential isolation, filesystem restrictions) must behave identically on both platforms. Platform-specific implementations would require separate security reviews, separate penetration testing, and continuous validation that policy enforcement is equivalent. The container abstraction provides a single implementation of the security boundary. From 8a5c3e43fc3e8936a9341505a19259e9f8268c45 Mon Sep 17 00:00:00 2001 From: Greg Allen Date: Mon, 11 May 2026 10:41:59 -0400 Subject: [PATCH 15/15] Fix broken ADR reference and add missing ADR-0030 citation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix broken link in agent-execution-environment.md: ADR-0029 → ADR-0036 - Add ADR-0030 reference to ADR-0036 References section Addresses High and Medium severity findings from fullsend-ai-review. Co-Authored-By: Claude Sonnet 4.5 --- docs/ADRs/0036-agent-execution-sandbox.md | 1 + docs/plans/agent-execution-environment.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/ADRs/0036-agent-execution-sandbox.md b/docs/ADRs/0036-agent-execution-sandbox.md index de6ca33a17..243d226a59 100644 --- a/docs/ADRs/0036-agent-execution-sandbox.md +++ b/docs/ADRs/0036-agent-execution-sandbox.md @@ -289,5 +289,6 @@ The implementation document is structured for iterative evolution as the sandbox - ADR-0017: Credential isolation for sandboxed agents (zero credentials in sandbox) - ADR-0025: Provider credential delivery (OpenShell providers for credential injection) - [ADR-0028: GitLab Support Architecture](0028-gitlab-support.md) (dispatch pipelines, explicitly deferred agent execution environment) +- ADR-0030: OpenShell sandbox interaction model (defines the agent-harness communication protocol) - [agent-infrastructure.md](../problems/agent-infrastructure.md): Infrastructure layer exploration, SIG Agent Sandbox evaluation - [OpenShell](https://github.com/NVIDIA/OpenShell): Sandbox runtime with L7 network policy enforcement diff --git a/docs/plans/agent-execution-environment.md b/docs/plans/agent-execution-environment.md index 4b61a7dcec..b75ad3f25f 100644 --- a/docs/plans/agent-execution-environment.md +++ b/docs/plans/agent-execution-environment.md @@ -919,7 +919,7 @@ spec: ## References -- [ADR-0029: Agent Execution Sandbox Architecture](../ADRs/0029-agent-execution-sandbox.md) +- [ADR-0036: Agent Execution Sandbox Architecture](../ADRs/0036-agent-execution-sandbox.md) - [ADR-0017: Credential Isolation for Sandboxed Agents](../ADRs/0017-credential-isolation-for-sandboxed-agents.md) - [ADR-0025: Provider Credential Delivery](../ADRs/0025-provider-credential-delivery-for-sandboxed-agents.md) - [ADR-0028: GitLab Support Architecture](../ADRs/0028-gitlab-support.md)