Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ bun apps/worker/src/index.ts # pg-boss consumers + engine
cd apps/web && bun run dev # :5173, proxies /api → :3000
```

The env vars can instead live in a git-ignored `.env.local` at the repo root (Bun loads it natively; shell exports take precedence). Prefer that for anything beyond a throwaway session — `AGRIPPA_SECRET_KEY` encrypts stored dev credentials, and regenerating it per shell orphans them. See [CONTRIBUTING.md](CONTRIBUTING.md).

**Tests need Postgres.** Integration suites hit `TEST_DATABASE_URL` (default `postgres://localhost:5432/agrippa_test`) and **skip themselves** when it's unreachable — "N tests skipped" locally is not the same as green. CI provides postgres+redis services. Redis is optional everywhere: SSE falls back to DB polling, cancellation to step-boundary flag checks.

## Monorepo map & dependency direction
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ All notable changes to Agrippa are documented here. The format follows

## [Unreleased]

### Added

- **VM deployment without Docker** — `infra/vm/` ships an idempotent Ubuntu 22.04/24.04 installer (Bun, Postgres 17 via PGDG, optional Redis 7, `agrippa` system user, env file with generated secrets), a git-pull `deploy.sh`, and hardened systemd units. The api still migrates + seeds on boot; the worker gates its start on the api's `/healthz` so it never runs against a stale schema. The worker unit deliberately leaves namespaces unrestricted (bubblewrap needs them — the sandbox degrades silently otherwise). Docker Compose and local dev mode are unchanged.

### Security

- **Executor isolation seam** — one enforceable place (`packages/executor-core/isolation.ts`) decides every tool call and scrubs the subprocess environment. Read-only workspaces now actually deny shell and confine writes to the artifact directory; read-write workspaces confine writes to the workspace with a boundary-safe check (the previous `startsWith` let a sibling `<workspace>-evil` path through, and `Bash` bypassed the check entirely). **Reads (Read/Grep/Glob) are confined to the workspace too**, so the agent can't read `/proc/self/environ`, another run's directory, or the shared artifact store. The SDK subprocess environment is **allow-listed** — only the SDK auth variables and a fixed set of system essentials pass through, so platform secrets, DSNs, and injection vectors like `NODE_OPTIONS` are all dropped — with the OS `sandbox` enabled where available (bubblewrap installed in the worker image), `strictMcpConfig`, and repo-supplied `.claude`/`.mcp.json` removed after checkout so a checked-out repository can't inject hooks or permission overrides. The worker image runs as a non-root user with `/app` kept root-owned.
Expand All @@ -16,6 +20,7 @@ All notable changes to Agrippa are documented here. The format follows

### Fixed

- **Dev-mode sign-in through the Vite proxy** — better-auth trusts only its `baseURL` origin (`http://localhost:3000`), so auth POSTs from the dev SPA on `:5173` were rejected with 403 `INVALID_ORIGIN`. The dev proxy now rewrites the `Origin` header to the API's own, completing the same-origin simulation of the production deployment (ADR-0001) instead of loosening the API's origin check.
- **Crash recovery for no-retry steps** — a worker that died mid-step no longer silently skips (or spuriously fails) a step without template retries; the crashed attempt no longer consumes the retry budget, and the executor session is carried onto the recovery attempt so resume works.
- **Atomic run lifecycle** — one `finalizeRun` owns every terminal transition: a single transaction does the status CAS (requiring `cancel_requested = false` for a success, so a late cancel wins atomically), finishedAt/totals, and the terminal event. Both the engine and the worker's retry-exhaustion path use it, so a run is never half-finalized and a queued run whose setup threw transitions `queued → failed` (now legal) with a terminal event instead of stranding. Event sequence numbers come from an atomic per-run counter (`runs.next_event_seq`), and approval decisions are CAS on `pending` with the sweeper re-enqueuing any run left paused by a lost resume enqueue.
- **Quota accounting** — the engine now counts the same monthly window as the submit gate, excludes the run's own spend from the headroom it checks (no double-count on resume), re-reads project usage at each step boundary so concurrent runs can't jointly overspend, and restores per-phase spend on resume so per-phase budgets aren't reset by a crash.
Expand Down
12 changes: 8 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,19 @@ Requirements: **Bun ≥ 1.3**, **PostgreSQL ≥ 17** (local install or `docker c
```sh
bun install
createdb agrippa && createdb agrippa_test # or let compose provide them
export DATABASE_URL=postgres://localhost:5432/agrippa
export AGRIPPA_SECRET_KEY=$(openssl rand -base64 32)
export AGRIPPA_EXECUTOR=fake # token-free demo executor
cat > .env.local <<EOF
DATABASE_URL=postgres://localhost:5432/agrippa
AGRIPPA_SECRET_KEY=$(openssl rand -base64 32)
AGRIPPA_EXECUTOR=fake # token-free demo executor
EOF
bun apps/api/src/index.ts # migrates + seeds on boot
bun apps/worker/src/index.ts
cd apps/web && bun run dev # http://localhost:5173
```

The first account you sign up becomes the org admin. Real agent runs need `AGRIPPA_EXECUTOR=claude-agent-sdk` plus `ANTHROPIC_API_KEY` on the worker.
Bun loads `.env` / `.env.local` natively from the directory you start it in — no dotenv, no `export`s — and `.env.local` is git-ignored, so secrets can't be committed. Create it **once and keep it**: `AGRIPPA_SECRET_KEY` encrypts credentials you store in dev (repo tokens, MCP auth), and a fresh key per shell — the old `export $(openssl rand …)` workflow — silently orphans them. Plain `export`s still work if you prefer; they take precedence over the file.

The first account you sign up becomes the org admin. Real agent runs need `AGRIPPA_EXECUTOR=claude-agent-sdk` plus `ANTHROPIC_API_KEY` (worker env).

## Quality gates

Expand Down
19 changes: 14 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,17 @@ M1 implemented — all three layers work end to end. See the [M1 plan](docs/plan
```sh
bun install
docker compose -f infra/docker-compose.dev.yml up -d # postgres + redis (or use local installs)
export DATABASE_URL=postgres://localhost:5432/agrippa
export AGRIPPA_SECRET_KEY=$(openssl rand -base64 32)
export AGRIPPA_EXECUTOR=fake # token-free demo executor
cat > .env.local <<EOF # created once; Bun loads it automatically (git-ignored)
DATABASE_URL=postgres://localhost:5432/agrippa
AGRIPPA_SECRET_KEY=$(openssl rand -base64 32)
AGRIPPA_EXECUTOR=fake
EOF
bun apps/api/src/index.ts # api :3000 (migrates + seeds on boot)
bun apps/worker/src/index.ts # worker
cd apps/web && bun run dev # SPA :5173 (proxies /api → :3000)
```

Sign up (the first user becomes org admin), create a project, grant models/skills under Settings → Resources, then submit a task from the catalog. Set `AGRIPPA_EXECUTOR=claude-agent-sdk` and `ANTHROPIC_API_KEY` for real runs.
Sign up (the first user becomes org admin), create a project, grant models/skills under Settings → Resources, then submit a task from the catalog. For real runs set `AGRIPPA_EXECUTOR=claude-agent-sdk` and `ANTHROPIC_API_KEY` in `.env.local` — and keep the file: `AGRIPPA_SECRET_KEY` encrypts credentials you store, so regenerating it orphans them (see [CONTRIBUTING.md](CONTRIBUTING.md)).

**Self-hosted** (Docker):

Expand All @@ -40,6 +42,13 @@ cp infra/env/.env.example infra/env/.env # fill in the secrets
docker compose -f infra/docker-compose.yml --env-file infra/env/.env up -d
```

**Self-hosted** (VM, systemd — no Docker; Ubuntu 22.04/24.04):

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # idempotent; updates: infra/vm/deploy.sh
```

**Quality gate**: `bun run check && bun test` (same commands CI runs).

## Documentation
Expand All @@ -51,7 +60,7 @@ docker compose -f infra/docker-compose.yml --env-file infra/env/.env up -d

## Tech stack

TypeScript · Bun workspaces monorepo · Hono (REST + SSE) · Vite + React + TailwindCSS + shadcn/ui · Drizzle + Postgres · pg-boss · Redis (pubsub) · better-auth · i18next (en / zh-CN) · Claude Agent SDK (first executor) · Docker Compose (self-hosted)
TypeScript · Bun workspaces monorepo · Hono (REST + SSE) · Vite + React + TailwindCSS + shadcn/ui · Drizzle + Postgres · pg-boss · Redis (pubsub) · better-auth · i18next (en / zh-CN) · Claude Agent SDK (first executor) · Docker Compose or systemd VM (self-hosted)

---

Expand Down
12 changes: 11 additions & 1 deletion apps/web/vite.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,17 @@ export default defineConfig({
},
server: {
proxy: {
"/api": "http://localhost:3000",
"/api": {
target: "http://localhost:3000",
// Dev simulates the production same-origin deployment (ADR-0001: the
// API serves the SPA, no CORS). Rewrite Origin to the API's own, or
// better-auth's CSRF check 403s every auth POST with INVALID_ORIGIN.
configure(proxy) {
proxy.on("proxyReq", (proxyReq) => {
proxyReq.setHeader("origin", "http://localhost:3000");
});
},
},
},
},
});
30 changes: 28 additions & 2 deletions docs/design/08-deployment.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
# 08 — Deployment (Self-Hosted, Docker Compose)
# 08 — Deployment (Self-Hosted)

> Status: draft for review · Last updated: 2026-07-17

M1 ships as a single-org, self-hosted Docker Compose stack. SaaS multi-tenancy is a non-goal but the schema is ready (`org_id` everywhere).
M1 ships single-org and self-hosted. SaaS multi-tenancy is a non-goal but the schema is ready (`org_id` everywhere). Three ways to run it:

1. **Development** — bare-metal Bun + local Postgres; `infra/docker-compose.dev.yml` optionally provides dependencies only (see the README quick start).
2. **Docker Compose** — the containerized production stack below.
3. **VM (systemd, no Docker)** — same processes as Compose, installed directly on an Ubuntu host via `infra/vm/` (see [VM deployment](#vm-deployment-systemd-no-docker)).

## Topology

Expand Down Expand Up @@ -45,6 +49,28 @@ Secrets policy: provider API keys and the master key live only in `api`/`worker`
- **TLS / ingress**: out of scope; operators front the stack with their own reverse proxy. SSE requires the proxy to disable response buffering for `/api/v1/runs/*/events`.
- **Health**: `GET /healthz` (api: DB ping). A per-worker heartbeat row for the admin UI is deferred past M1.

## VM deployment (systemd, no Docker)

`infra/vm/` codifies the same topology on a single Ubuntu 22.04/24.04 host: `install.sh` (idempotent bootstrap: Bun, Postgres 17 via PGDG, optional Redis 7, `agrippa` system user, env file with generated secrets, units), `deploy.sh` (update: `git pull --ff-only` → `bun install --frozen-lockfile` → SPA build → restart), `agrippa-api.service` / `agrippa-worker.service`, `env.example`, `nginx.conf.example`.

| Piece | Location | Compose equivalent |
|---|---|---|
| Repo + build | `/opt/agrippa` (root-owned; services read-only by permissions) | root-owned `/app` in the images |
| Runtime user | `agrippa` system user (both services) | `bun` user in both containers |
| Run workspaces | `/var/lib/agrippa/runs` (`WORKSPACE_ROOT`) | `workspaces` volume |
| Artifact store | `/var/lib/agrippa/artifacts` (`ARTIFACT_STORAGE_ROOT`) | `artifacts` volume |
| Config | `/etc/agrippa/agrippa.env` (root:agrippa 0640, one `EnvironmentFile=` for both units) | compose `env_file` |
| SPA | built on deploy, served by api via `AGRIPPA_WEB_DIST` | built at image build time |

Boot/upgrade ordering: postgres → api (migrates + seeds on boot) → worker. The worker never migrates; its `ExecStartPre` polls the api's `/healthz` (up to 120 s) so a first boot or an upgrade with new migrations can't run against a stale schema, and `deploy.sh` restarts the api and waits for `/healthz` before restarting the worker.

Sandboxing: the worker unit's hardening is deliberately lighter than the api's — bubblewrap needs user/mount/pid/net namespaces, so `RestrictNamespaces=`/`SystemCallFilter=` must stay off (the SDK sandbox degrades *silently* without them). On Ubuntu 24.04, verify `apparmor_restrict_unprivileged_userns` doesn't block bwrap: `sudo -u agrippa bwrap --unshare-all --ro-bind / / /bin/true`.

Two accepted deviations from Compose (documented, not accidental):

- **Artifacts are not read-only to the api.** Compose mounts the artifact volume `:ro` in the api container; on the VM both services share the `agrippa` user, so the api could write artifacts. Splitting users would require group-permission choreography on every worker-written file for little gain on a single-org box.
- **One env file for both services.** The api process can read `ANTHROPIC_API_KEY`, which Compose scopes to the worker. Operators who care can split a `worker.env` and point the worker unit's `EnvironmentFile=` at it.

## Release Pipeline

`.github/workflows/release.yml`: on tag `v*` → build `agrippa-api` and `agrippa-worker` images → push to GHCR → attach compose bundle to the GitHub release. Acceptance check for M1.6: fresh machine, `docker compose up`, complete a bug-fix run end-to-end through the browser.
25 changes: 25 additions & 0 deletions docs/manual/en/01-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,31 @@ Open `http://localhost:3000`. The stack is four services: **api** (also serves t

**Demo mode**: set `AGRIPPA_EXECUTOR=fake` and leave `ANTHROPIC_API_KEY` empty — every task type runs end to end with a token-free demo executor that produces placeholder artifacts. Ideal for evaluating the platform before spending tokens.

## Deploy on a VM (systemd, no Docker)

Requirements: Ubuntu 22.04/24.04 LTS, ~2 GB RAM, root access.

```sh
sudo git clone https://github.com/ainaive/agrippa /opt/agrippa
sudo /opt/agrippa/infra/vm/install.sh # add --skip-redis to omit Redis
```

The installer is idempotent and sets up everything on one box:

- Bun, plus the worker's OS dependencies (`git`, `ripgrep`, `bubblewrap` for the agent sandbox)
- PostgreSQL 17 (PGDG) and Redis 7 (optional — without it, live streams fall back to polling)
- an `agrippa` system user, data directories under `/var/lib/agrippa`
- `/etc/agrippa/agrippa.env` with generated secrets — **back up `AGRIPPA_SECRET_KEY`**
- `agrippa-api` + `agrippa-worker` systemd units, then the first build and start

Open `http://<host>:3000`. The demo-mode note above applies here too (`AGRIPPA_EXECUTOR=fake` in `/etc/agrippa/agrippa.env`). To update later:

```sh
sudo /opt/agrippa/infra/vm/deploy.sh # pull → build → restart (api first, then worker)
```

See [Operations](06-operations.md) for logs, backup, and troubleshooting on a VM.

## Run from source (development)

See the [README quick start](../../../README.md#getting-started): Bun ≥ 1.3 + local Postgres, three processes (`api`, `worker`, `web`).
Expand Down
22 changes: 19 additions & 3 deletions docs/manual/en/06-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,20 @@
| `postgres` | System of record | Also carries the job queue (pg-boss) — no separate broker |
| `redis` | Live-event fan-out only | **Disposable**: if it's down, live streams degrade to replay/polling; correctness is unaffected |

## VM (systemd) deployment

The same stack installed by `infra/vm/install.sh` on one Ubuntu host (no Docker; see [Getting Started](01-getting-started.md#deploy-on-a-vm-systemd-no-docker)):

| Piece | Where |
|---|---|
| Services | `agrippa-api.service`, `agrippa-worker.service` (Postgres and Redis run as regular system services) |
| Logs | `journalctl -u agrippa-api -f` · `journalctl -u agrippa-worker -f` |
| Code + SPA build | `/opt/agrippa` (root-owned) |
| Config | `/etc/agrippa/agrippa.env` — one file for both services |
| Run workspaces / artifacts | `/var/lib/agrippa/runs` · `/var/lib/agrippa/artifacts` |

Updates: `sudo /opt/agrippa/infra/vm/deploy.sh` — pulls (`--ff-only`), installs with the frozen lockfile, rebuilds the SPA, restarts the api, waits for `/healthz` (migrations apply on api boot), then restarts the worker. Config changes take effect with `sudo systemctl restart agrippa-api agrippa-worker`.

## Configuration reference

Documented in `infra/env/.env.example`; the full set:
Expand All @@ -35,13 +49,13 @@ Documented in `infra/env/.env.example`; the full set:

## Backup — three things

1. The **postgres volume** (`pgdata`) — schedule `pg_dump` per your policy.
2. The **work volume** (`workdata`) — holds large artifacts; losing it loses downloads over 64 KB (metadata and small artifacts survive in Postgres).
1. The **database** — Compose: the `pgdata` volume; VM: `pg_dump agrippa` — schedule per your policy.
2. The **artifact store** — Compose: the `artifacts` volume; VM: `/var/lib/agrippa/artifacts`. Losing it loses downloads over 64 KB (metadata and small artifacts survive in Postgres).
3. **`AGRIPPA_SECRET_KEY`** — without it, every stored git token and MCP credential is unrecoverable. Redis needs no backup.

## Upgrades & scaling

Pull new images and `docker compose up -d`. The api migrates on boot under an advisory lock, so rolling multiple replicas is safe. Draining workers is safe too: a killed worker's in-flight runs stay `running`, the queue retries them, and the engine **resumes step-granularly** — completed steps are never re-executed and cost is never double-counted. Scale run throughput with `WORKER_REPLICAS` × `WORKER_SLOTS`.
Pull new images and `docker compose up -d` (VM: `sudo /opt/agrippa/infra/vm/deploy.sh`, which restarts the api first — see the VM section above). The api migrates on boot under an advisory lock, so rolling multiple replicas is safe. Draining workers is safe too: a killed worker's in-flight runs stay `running`, the queue retries them, and the engine **resumes step-granularly** — completed steps are never re-executed and cost is never double-counted. Scale run throughput with `WORKER_REPLICAS` × `WORKER_SLOTS`.

Reverse proxy note: **disable response buffering** for `/api/v1/runs/*/events` (SSE) — e.g. `proxy_buffering off;` in nginx — or live progress will arrive in bursts.

Expand All @@ -59,3 +73,5 @@ Reverse proxy note: **disable response buffering** for `/api/v1/runs/*/events` (
| Checkout fails for a private repo | The repo connection's token is missing/expired — re-add it under Settings → Repositories (tokens are write-only; re-enter, don't "view"). |
| Need to inspect what an agent actually did on disk | Set `AGRIPPA_KEEP_WORKSPACES=1` on the worker and re-run; workspaces persist under `WORKSPACE_ROOT/<runId>`. |
| `healthz` returns 503 | The api can't reach Postgres — check `DATABASE_URL` and the postgres service. |
| (VM) worker stuck in "activating" | Its `ExecStartPre` is waiting for the api's `/healthz` (up to 120 s) — check `journalctl -u agrippa-api` for why the api isn't healthy. |
| (VM) agent commands fail, or sandboxing is suspect on Ubuntu 24.04 | AppArmor's `apparmor_restrict_unprivileged_userns` can block bubblewrap — and without bwrap the sandbox degrades **silently**. Probe with `sudo -u agrippa bwrap --unshare-all --ro-bind / / /bin/true`; if it fails, allow unprivileged user namespaces (or install a bwrap AppArmor profile) and restart the worker. |
Loading
Loading