Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ All notable changes to Agrippa are documented here. The format follows

## [Unreleased]

### Added

- **VM deployment without Docker** — `infra/vm/` ships an idempotent Ubuntu 22.04/24.04 installer (Bun, Postgres 17 via PGDG, optional Redis 7, `agrippa` system user, env file with generated secrets), a git-pull `deploy.sh`, and hardened systemd units. The api still migrates + seeds on boot; the worker gates its start on the api's `/healthz` so it never runs against a stale schema. The worker unit deliberately leaves namespaces unrestricted (bubblewrap needs them — the sandbox degrades silently otherwise). Docker Compose and local dev mode are unchanged.

### Security

- **Executor isolation seam** — one enforceable place (`packages/executor-core/isolation.ts`) decides every tool call and scrubs the subprocess environment. Read-only workspaces now actually deny shell and confine writes to the artifact directory; read-write workspaces confine writes to the workspace with a boundary-safe check (the previous `startsWith` let a sibling `<workspace>-evil` path through, and `Bash` bypassed the check entirely). **Reads (Read/Grep/Glob) are confined to the workspace too**, so the agent can't read `/proc/self/environ`, another run's directory, or the shared artifact store. The SDK subprocess environment is **allow-listed** — only the SDK auth variables and a fixed set of system essentials pass through, so platform secrets, DSNs, and injection vectors like `NODE_OPTIONS` are all dropped — with the OS `sandbox` enabled where available (bubblewrap installed in the worker image), `strictMcpConfig`, and repo-supplied `.claude`/`.mcp.json` removed after checkout so a checked-out repository can't inject hooks or permission overrides. The worker image runs as a non-root user with `/app` kept root-owned.
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,13 @@ cp infra/env/.env.example infra/env/.env # fill in the secrets
docker compose -f infra/docker-compose.yml --env-file infra/env/.env up -d
```

**Self-hosted** (VM, systemd — no Docker; Ubuntu 22.04/24.04):

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # idempotent; updates: infra/vm/deploy.sh
```

**Quality gate**: `bun run check && bun test` (same commands CI runs).

## Documentation
Expand All @@ -51,7 +58,7 @@ docker compose -f infra/docker-compose.yml --env-file infra/env/.env up -d

## Tech stack

TypeScript · Bun workspaces monorepo · Hono (REST + SSE) · Vite + React + TailwindCSS + shadcn/ui · Drizzle + Postgres · pg-boss · Redis (pubsub) · better-auth · i18next (en / zh-CN) · Claude Agent SDK (first executor) · Docker Compose (self-hosted)
TypeScript · Bun workspaces monorepo · Hono (REST + SSE) · Vite + React + TailwindCSS + shadcn/ui · Drizzle + Postgres · pg-boss · Redis (pubsub) · better-auth · i18next (en / zh-CN) · Claude Agent SDK (first executor) · Docker Compose or systemd VM (self-hosted)

---

Expand Down
30 changes: 28 additions & 2 deletions docs/design/08-deployment.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# 08 — Deployment (Self-Hosted, Docker Compose)
# 08 — Deployment (Self-Hosted)

> Status: draft for review · Last updated: 2026-07-17

M1 ships as a single-org, self-hosted Docker Compose stack. SaaS multi-tenancy is a non-goal but the schema is ready (`org_id` everywhere).
M1 ships single-org and self-hosted. SaaS multi-tenancy is a non-goal but the schema is ready (`org_id` everywhere). Three ways to run it:

1. **Development** — bare-metal Bun + local Postgres; `infra/docker-compose.dev.yml` optionally provides dependencies only (see the README quick start).
2. **Docker Compose** — the containerized production stack below.
3. **VM (systemd, no Docker)** — same processes as Compose, installed directly on an Ubuntu host via `infra/vm/` (see [VM deployment](#vm-deployment-systemd-no-docker)).

## Topology

Expand Down Expand Up @@ -45,6 +49,28 @@ Secrets policy: provider API keys and the master key live only in `api`/`worker`
- **TLS / ingress**: out of scope; operators front the stack with their own reverse proxy. SSE requires the proxy to disable response buffering for `/api/v1/runs/*/events`.
- **Health**: `GET /healthz` (api: DB ping). A per-worker heartbeat row for the admin UI is deferred past M1.

## VM deployment (systemd, no Docker)

`infra/vm/` codifies the same topology on a single Ubuntu 22.04/24.04 host: `install.sh` (idempotent bootstrap: Bun, Postgres 17 via PGDG, optional Redis 7, `agrippa` system user, env file with generated secrets, units), `deploy.sh` (update: `git pull --ff-only` → `bun install --frozen-lockfile` → SPA build → restart), `agrippa-api.service` / `agrippa-worker.service`, `env.example`, `nginx.conf.example`.

| Piece | Location | Compose equivalent |
|---|---|---|
| Repo + build | `/opt/agrippa` (root-owned; services read-only by permissions) | root-owned `/app` in the images |
| Runtime user | `agrippa` system user (both services) | `bun` user in both containers |
| Run workspaces | `/var/lib/agrippa/runs` (`WORKSPACE_ROOT`) | `workspaces` volume |
| Artifact store | `/var/lib/agrippa/artifacts` (`ARTIFACT_STORAGE_ROOT`) | `artifacts` volume |
| Config | `/etc/agrippa/agrippa.env` (root:agrippa 0640, one `EnvironmentFile=` for both units) | compose `env_file` |
| SPA | built on deploy, served by api via `AGRIPPA_WEB_DIST` | built at image build time |

Boot/upgrade ordering: postgres → api (migrates + seeds on boot) → worker. The worker never migrates; its `ExecStartPre` polls the api's `/healthz` (up to 120 s) so a first boot or an upgrade with new migrations can't run against a stale schema, and `deploy.sh` restarts the api and waits for `/healthz` before restarting the worker.

Sandboxing: the worker unit's hardening is deliberately lighter than the api's — bubblewrap needs user/mount/pid/net namespaces, so `RestrictNamespaces=`/`SystemCallFilter=` must stay off (the SDK sandbox degrades *silently* without them). On Ubuntu 24.04, verify `apparmor_restrict_unprivileged_userns` doesn't block bwrap: `sudo -u agrippa bwrap --unshare-all --ro-bind / / /bin/true`.

Two accepted deviations from Compose (documented, not accidental):

- **Artifacts are not read-only to the api.** Compose mounts the artifact volume `:ro` in the api container; on the VM both services share the `agrippa` user, so the api could write artifacts. Splitting users would require group-permission choreography on every worker-written file for little gain on a single-org box.
- **One env file for both services.** The api process can read `ANTHROPIC_API_KEY`, which Compose scopes to the worker. Operators who care can split a `worker.env` and point the worker unit's `EnvironmentFile=` at it.

## Release Pipeline

`.github/workflows/release.yml`: on tag `v*` → build `agrippa-api` and `agrippa-worker` images → push to GHCR → attach compose bundle to the GitHub release. Acceptance check for M1.6: fresh machine, `docker compose up`, complete a bug-fix run end-to-end through the browser.
25 changes: 25 additions & 0 deletions docs/manual/en/01-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,31 @@ Open `http://localhost:3000`. The stack is four services: **api** (also serves t

**Demo mode**: set `AGRIPPA_EXECUTOR=fake` and leave `ANTHROPIC_API_KEY` empty — every task type runs end to end with a token-free demo executor that produces placeholder artifacts. Ideal for evaluating the platform before spending tokens.

## Deploy on a VM (systemd, no Docker)

Requirements: Ubuntu 22.04/24.04 LTS, ~2 GB RAM, root access.

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # add --skip-redis to omit Redis
```

The installer is idempotent and sets up everything on one box:

- Bun, plus the worker's OS dependencies (`git`, `ripgrep`, `bubblewrap` for the agent sandbox)
- PostgreSQL 17 (PGDG) and Redis 7 (optional — without it, live streams fall back to polling)
- an `agrippa` system user, data directories under `/var/lib/agrippa`
- `/etc/agrippa/agrippa.env` with generated secrets — **back up `AGRIPPA_SECRET_KEY`**
- `agrippa-api` + `agrippa-worker` systemd units, then the first build and start

Open `http://<host>:3000`. The demo-mode note above applies here too (`AGRIPPA_EXECUTOR=fake` in `/etc/agrippa/agrippa.env`). To update later:

```sh
sudo /opt/agrippa/infra/vm/deploy.sh # pull → build → restart (api first, then worker)
```

See [Operations](06-operations.md) for logs, backup, and troubleshooting on a VM.

## Run from source (development)

See the [README quick start](../../../README.md#getting-started): Bun ≥ 1.3 + local Postgres, three processes (`api`, `worker`, `web`).
Expand Down
22 changes: 19 additions & 3 deletions docs/manual/en/06-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,20 @@
| `postgres` | System of record | Also carries the job queue (pg-boss) — no separate broker |
| `redis` | Live-event fan-out only | **Disposable**: if it's down, live streams degrade to replay/polling; correctness is unaffected |

## VM (systemd) deployment

The same stack installed by `infra/vm/install.sh` on one Ubuntu host (no Docker; see [Getting Started](01-getting-started.md#deploy-on-a-vm-systemd-no-docker)):

| Piece | Where |
|---|---|
| Services | `agrippa-api.service`, `agrippa-worker.service` (Postgres and Redis run as regular system services) |
| Logs | `journalctl -u agrippa-api -f` · `journalctl -u agrippa-worker -f` |
| Code + SPA build | `/opt/agrippa` (root-owned) |
| Config | `/etc/agrippa/agrippa.env` — one file for both services |
| Run workspaces / artifacts | `/var/lib/agrippa/runs` · `/var/lib/agrippa/artifacts` |

Updates: `sudo /opt/agrippa/infra/vm/deploy.sh` — pulls (`--ff-only`), installs with the frozen lockfile, rebuilds the SPA, restarts the api, waits for `/healthz` (migrations apply on api boot), then restarts the worker. Config changes take effect with `sudo systemctl restart agrippa-api agrippa-worker`.

## Configuration reference

Documented in `infra/env/.env.example`; the full set:
Expand All @@ -35,13 +49,13 @@ Documented in `infra/env/.env.example`; the full set:

## Backup — three things

1. The **postgres volume** (`pgdata`) — schedule `pg_dump` per your policy.
2. The **work volume** (`workdata`) — holds large artifacts; losing it loses downloads over 64 KB (metadata and small artifacts survive in Postgres).
1. The **database** — Compose: the `pgdata` volume; VM: `pg_dump agrippa` — schedule per your policy.
2. The **artifact store** — Compose: the `artifacts` volume; VM: `/var/lib/agrippa/artifacts`. Losing it loses downloads over 64 KB (metadata and small artifacts survive in Postgres).
3. **`AGRIPPA_SECRET_KEY`** — without it, every stored git token and MCP credential is unrecoverable. Redis needs no backup.

## Upgrades & scaling

Pull new images and `docker compose up -d`. The api migrates on boot under an advisory lock, so rolling multiple replicas is safe. Draining workers is safe too: a killed worker's in-flight runs stay `running`, the queue retries them, and the engine **resumes step-granularly** — completed steps are never re-executed and cost is never double-counted. Scale run throughput with `WORKER_REPLICAS` × `WORKER_SLOTS`.
Pull new images and `docker compose up -d` (VM: `sudo /opt/agrippa/infra/vm/deploy.sh`, which restarts the api first — see the VM section above). The api migrates on boot under an advisory lock, so rolling multiple replicas is safe. Draining workers is safe too: a killed worker's in-flight runs stay `running`, the queue retries them, and the engine **resumes step-granularly** — completed steps are never re-executed and cost is never double-counted. Scale run throughput with `WORKER_REPLICAS` × `WORKER_SLOTS`.

Reverse proxy note: **disable response buffering** for `/api/v1/runs/*/events` (SSE) — e.g. `proxy_buffering off;` in nginx — or live progress will arrive in bursts.

Expand All @@ -59,3 +73,5 @@ Reverse proxy note: **disable response buffering** for `/api/v1/runs/*/events` (
| Checkout fails for a private repo | The repo connection's token is missing/expired — re-add it under Settings → Repositories (tokens are write-only; re-enter, don't "view"). |
| Need to inspect what an agent actually did on disk | Set `AGRIPPA_KEEP_WORKSPACES=1` on the worker and re-run; workspaces persist under `WORKSPACE_ROOT/<runId>`. |
| `healthz` returns 503 | The api can't reach Postgres — check `DATABASE_URL` and the postgres service. |
| (VM) worker stuck in "activating" | Its `ExecStartPre` is waiting for the api's `/healthz` (up to 120 s) — check `journalctl -u agrippa-api` for why the api isn't healthy. |
| (VM) agent commands fail, or sandboxing is suspect on Ubuntu 24.04 | AppArmor's `apparmor_restrict_unprivileged_userns` can block bubblewrap — and without bwrap the sandbox degrades **silently**. Probe with `sudo -u agrippa bwrap --unshare-all --ro-bind / / /bin/true`; if it fails, allow unprivileged user namespaces (or install a bwrap AppArmor profile) and restart the worker. |
25 changes: 25 additions & 0 deletions docs/manual/zh-CN/01-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,31 @@ docker compose -f infra/docker-compose.yml --env-file infra/env/.env up -d

**演示模式**:设置 `AGRIPPA_EXECUTOR=fake` 并将 `ANTHROPIC_API_KEY` 留空——所有任务类型都会由零 Token 消耗的演示执行器端到端跑通,产出占位产出物。适合在花费 Token 之前评估平台。

## 部署到虚拟机(systemd,无需 Docker)

要求:Ubuntu 22.04/24.04 LTS,约 2 GB 内存,root 权限。

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # 加 --skip-redis 可不装 Redis
```

安装脚本幂等(可重复执行),在同一台机器上完成全部准备:

- Bun,以及 worker 的系统依赖(`git`、`ripgrep`、用于智能体沙箱的 `bubblewrap`)
- PostgreSQL 17(PGDG)与 Redis 7(可选——没有它实时流会降级为轮询)
- `agrippa` 系统用户,数据目录位于 `/var/lib/agrippa`
- `/etc/agrippa/agrippa.env`,密钥自动生成——**务必备份 `AGRIPPA_SECRET_KEY`**
- `agrippa-api` + `agrippa-worker` systemd 服务,随后完成首次构建并启动

打开 `http://<主机>:3000`。上文的演示模式同样适用(在 `/etc/agrippa/agrippa.env` 中设置 `AGRIPPA_EXECUTOR=fake`)。后续更新:

```sh
sudo /opt/agrippa/infra/vm/deploy.sh # 拉取 → 构建 → 重启(先 api 后 worker)
```

虚拟机上的日志、备份与故障排查见[运维指南](06-operations.md)。

## 源码运行(开发模式)

参见 [README 快速开始](../../../README.md#getting-started):Bun ≥ 1.3 + 本地 Postgres,启动三个进程(`api`、`worker`、`web`)。
Expand Down
22 changes: 19 additions & 3 deletions docs/manual/zh-CN/06-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,20 @@
| `postgres` | 事实来源 | 同时承载任务队列(pg-boss)——无需额外消息中间件 |
| `redis` | 仅用于实时事件分发 | **可丢弃**:宕机时实时流降级为回放/轮询,正确性不受影响 |

## 虚拟机(systemd)部署

由 `infra/vm/install.sh` 在单台 Ubuntu 主机上安装的同一套服务栈(无需 Docker;参见[快速开始](01-getting-started.md#部署到虚拟机systemd无需-docker)):

| 内容 | 位置 |
|---|---|
| 服务 | `agrippa-api.service`、`agrippa-worker.service`(Postgres 与 Redis 为常规系统服务) |
| 日志 | `journalctl -u agrippa-api -f` · `journalctl -u agrippa-worker -f` |
| 代码 + SPA 构建 | `/opt/agrippa`(root 所有) |
| 配置 | `/etc/agrippa/agrippa.env` ——两个服务共用一个文件 |
| 执行工作区 / 产出物 | `/var/lib/agrippa/runs` · `/var/lib/agrippa/artifacts` |

更新:`sudo /opt/agrippa/infra/vm/deploy.sh` ——拉取(`--ff-only`)、按锁定文件安装依赖、重建 SPA、重启 api、等待 `/healthz`(迁移在 api 启动时应用)、再重启 worker。修改配置后执行 `sudo systemctl restart agrippa-api agrippa-worker` 生效。

## 配置项参考

以 `infra/env/.env.example` 为准;完整清单:
Expand All @@ -35,13 +49,13 @@

## 备份——三样东西

1. **postgres 卷**(`pgdata`)——按你的策略定期 `pg_dump`。
2. **work 卷**(`workdata`)——存放大产出物;丢失后超过 64 KB 的下载不可恢复(元数据与小产出物在 Postgres 中仍在)。
1. **数据库** —— Compose:`pgdata` 卷;虚拟机:`pg_dump agrippa` ——按你的策略定期执行。
2. **产出物存储** —— Compose:`artifacts` 卷;虚拟机:`/var/lib/agrippa/artifacts`。丢失后超过 64 KB 的下载不可恢复(元数据与小产出物在 Postgres 中仍在)。
3. **`AGRIPPA_SECRET_KEY`** ——没有它,所有已存的 git 令牌和 MCP 凭证都无法解密。Redis 无需备份。

## 升级与扩容

拉取新镜像后 `docker compose up -d` 即可。api 在启动时于咨询锁下迁移,多副本滚动升级安全。worker 排空同样安全:被终止的 worker 上进行中的执行保持 `running`,队列会重试,引擎**按步骤粒度续跑**——已完成的步骤不会重跑,费用也不会重复计入。吞吐量 = `WORKER_REPLICAS` × `WORKER_SLOTS`。
拉取新镜像后 `docker compose up -d` 即可(虚拟机:`sudo /opt/agrippa/infra/vm/deploy.sh`,会先重启 api——见上文虚拟机一节)。api 在启动时于咨询锁下迁移,多副本滚动升级安全。worker 排空同样安全:被终止的 worker 上进行中的执行保持 `running`,队列会重试,引擎**按步骤粒度续跑**——已完成的步骤不会重跑,费用也不会重复计入。吞吐量 = `WORKER_REPLICAS` × `WORKER_SLOTS`。

反向代理注意:对 `/api/v1/runs/*/events`(SSE)**关闭响应缓冲**——如 nginx 的 `proxy_buffering off;`——否则实时进度会成批到达。

Expand All @@ -59,3 +73,5 @@
| 私有仓库检出失败 | 仓库连接的令牌缺失或过期——到 设置 → 代码仓库 重新录入(令牌只写不读,重新填写即可)。 |
| 想看智能体在磁盘上到底做了什么 | 给 worker 设置 `AGRIPPA_KEEP_WORKSPACES=1` 后重跑;工作区保留在 `WORKSPACE_ROOT/<runId>`。 |
| `healthz` 返回 503 | api 连不上 Postgres——检查 `DATABASE_URL` 与 postgres 服务。 |
| (虚拟机)worker 卡在「activating」 | 其 `ExecStartPre` 正在等待 api 的 `/healthz`(最长 120 秒)——用 `journalctl -u agrippa-api` 排查 api 为何不健康。 |
| (虚拟机)Ubuntu 24.04 上智能体命令失败,或怀疑沙箱失效 | AppArmor 的 `apparmor_restrict_unprivileged_userns` 可能拦截 bubblewrap——而没有 bwrap 时沙箱会**静默**降级。用 `sudo -u agrippa bwrap --unshare-all --ro-bind / / /bin/true` 探测;若失败,放行非特权用户命名空间(或为 bwrap 安装 AppArmor 配置文件)后重启 worker。 |
21 changes: 21 additions & 0 deletions infra/vm/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# infra/vm — deploy Agrippa on a VM without Docker

Ubuntu 22.04/24.04 LTS, everything on one box (Bun, Postgres 17, optional
Redis, api + worker under systemd, the API serving the built SPA).

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # idempotent bootstrap + first deploy
sudo /opt/agrippa/infra/vm/deploy.sh # later updates: pull → build → restart
```

| File | Purpose |
| --- | --- |
| `install.sh` | Idempotent bootstrap: packages, Postgres 17 (PGDG), Redis 7 (`--skip-redis` to omit), Bun, `agrippa` user, `/etc/agrippa/agrippa.env` with generated secrets, systemd units |
| `deploy.sh` | Update: `git pull --ff-only` → `bun install` → build SPA → restart api (migrates on boot, gated on `/healthz`) → restart worker |
| `agrippa-api.service` / `agrippa-worker.service` | systemd units (the worker's lighter hardening is deliberate — bubblewrap needs namespaces) |
| `env.example` | Template rendered to `/etc/agrippa/agrippa.env` on first install |
| `nginx.conf.example` | Reverse-proxy example — SSE route needs `proxy_buffering off` |

Full guide: [docs/manual/en/06-operations.md](../../docs/manual/en/06-operations.md) ·
design: [docs/design/08-deployment.md](../../docs/design/08-deployment.md)
Loading
Loading