Skip to content

feat: Rebuild the factory image on every merge to main - #13781

Merged
anthonyshew merged 4 commits into
mainfrom
factory-snapshot-workflow
Aug 22, 2026
Merged

anthonyshew merged 4 commits into
mainfrom
factory-snapshot-workflow

Conversation

@anthonyshew

Copy link
Copy Markdown
Contributor

What

Agents in apps/factory booted from vercel/eve:latest plus a shallow clone. The sandbox had no Rust toolchain, protoc, Cap'n Proto, Zig, LLD, pnpm 10.28.0, node_modules, or performance tooling, and its Eve revalidationKey was hardcoded to "turborepo-main-v1", so the template never rotated. .devcontainer/Dockerfile had most of that toolchain but was unused and had drifted three Rust nightlies, two Node majors, and two pnpm majors behind.

This adds the factory image: one specification for what an agent's sandbox contains, a snapshot rebuilt by the application on every merge to main, and both agent paths booting from it.

One specification

apps/factory/agent/lib/factory-image.ts owns the image. It pins versions to the values the repository and CI already use, emits idempotent provisioning phases, and fingerprints the whole thing so a changed pin rebuilds every image.

Tool Version Source of truth
Rust nightly-2026-05-22 + rustfmt, clippy rust-toolchain.toml
Node.js 24 root package.json engines
pnpm 10.28.0 root package.json packageManager
protoc 26.1 .github/actions/setup-protoc
Cap'n Proto 1.1.0 .github/actions/setup-capnproto
Zig 0.15.2 .github/actions/setup-zig

Plus build-essential, pkg-config, LLD (.cargo/config.toml links with -fuse-ld=lld), OpenSSL headers, jq, zstd, the workspace node_modules, a warm Cargo registry, and hyperfine, cargo-bloat, and twiggy for the performance skill. A final phase verifies every required tool, warns about missing optional ones, and writes a version manifest.

.devcontainer/Dockerfile was rewritten against the same pins, and a test fails when either drifts from rust-toolchain.toml, the root package.json, or the CI actions — so the local dev container and the agents' sandbox stay on one toolchain.

Rebuilt on every merge, without GitHub Actions

A push to main reaches POST /api/github/push, which verifies the GitHub HMAC signature and starts a Workflow run. The workflow creates a build sandbox, detaches the provisioning script inside it, polls the markers the script writes, snapshots the result, and publishes the snapshot id to a Blob-backed ledger. No step holds a function invocation open for the length of a build. When a published image already exists for the same toolchain the build boots from it, so a merge build only fast-forwards the checkout, refreshes dependencies, and recompiles.

Rapid merges

Resolved in the ledger rather than by racing:

  • Claiming a build cancels every build still in flight, records which build replaced it, stops its workflow run, and deletes its sandbox.
  • Every step re-reads the ledger before acting; a build that has lost can neither report progress nor publish a pointer.
  • Redelivered webhooks deduplicate onto the live build, a revision that is already published is skipped, and a build that stops reporting progress for 15 minutes is replaced instead of wedging its revision.

Consumers

  • Eve (agent/sandbox.ts) provisions its template from the same phases and boots from the published snapshot when one matches. Eve freezes revalidationKey at build time, so the template rotates when the fingerprint changes or a newer image is published; each session then fast-forwards its checkout to the current main.
  • Harness (agent/lib/harness-agent.ts) uses the snapshot as its sandbox source instead of a stock node24 runtime plus a clone, rotates its template with the image, and falls back to cloning when no image matches this deployment's toolchain.
  • The operator page shows the published snapshot, the toolchain fingerprint, warnings, and recent builds, and can rebuild from the current main head.

Validation

Every provisioning phase was run against a real vercel/eve:latest sandbox. All eleven phases pass through verification in about two minutes, with no warnings:

system-packages 10s · node 1s · pnpm 2s · rust 11s · protoc <1s · zig 5s
checkout 2s · node-modules 12s · cargo-registry 7s · performance-tools 45s · verify 1s

{"capnp":"Cap'n Proto version 1.1.0","node":"v24.17.0","pnpm":"10.28.0",
 "protoc":"libprotoc 26.1","rustc":"rustc 1.97.0-nightly (e96c36b6f 2026-05-21)",
 "zig":"0.15.2"}

That run also confirmed the image is Ubuntu 26.04 with apt, that commands run as root, and that ld.lld, hyperfine, cargo-bloat, and twiggy all land on PATH.

45 unit tests pass (pnpm test), covering the pinned versions against the files they mirror, the phase list and generated script, revision validation, progress parsing, the fingerprint, the ledger's supersede and publish rules, and webhook signature and event filtering. eve build, next build, tsc --noEmit, oxlint --deny-warnings, and oxfmt --check are clean.

Deployment notes

  • Requires the private Vercel Blob store the run registry already uses.
  • Set FACTORY_IMAGE_WEBHOOK_SECRET (falls back to GITHUB_WEBHOOK_SECRET) and deliver push to /api/github/push. Deployment Protection covers that path, so append the automation bypass token as a query parameter; the HMAC signature authenticates the delivery.
  • The first build after a toolchain change provisions the Eve template during the Vercel build, because Eve prewarms templates there. The phases that compile Rust are wrapped in timeouts so one bad upstream release cannot hold a deployment build open, and only the merge webhook asks for the warm cargo build.
  • Performance-tool installation failures are recorded as warnings rather than failing a build, so a broken upstream crate cannot stop an otherwise complete image from publishing. The warnings surface on the operator page.

🤖 Generated with Claude Code

Turborepo agents booted from `vercel/eve:latest` plus a shallow clone,
with no Rust toolchain, protoc, Cap'n Proto, Zig, LLD, pnpm 10.28.0,
`node_modules`, or performance tooling, and a hardcoded revalidation key
that never rotated. `.devcontainer/Dockerfile` carried most of that
toolchain but was unused and had drifted three Rust nightlies, two Node
majors, and two pnpm majors behind.

`agent/lib/factory-image.ts` is now the single source of truth for the
image: pinned versions taken from `rust-toolchain.toml`, the root
`package.json`, and the CI setup actions; idempotent provisioning phases;
a verification phase that fails when a tool is missing and writes a
version manifest; and a fingerprint over all of it that decides when a
rebuild is required.

A push to `main` reaches `POST /api/github/push`, which verifies the
GitHub HMAC signature and starts a Workflow run. That workflow detaches
the provisioning script inside a build sandbox, polls the markers the
script writes, snapshots the result, and publishes the snapshot id, so
no GitHub Actions job is involved and no step holds a function open for
the length of a build. Builds start from the previous image when the
toolchain fingerprint matches, so a merge only fast-forwards the
checkout, refreshes dependencies, and recompiles.

Rapid merges are resolved in the ledger rather than by racing. Claiming
a build cancels every build still in flight, stops its workflow run, and
deletes its sandbox; each step re-reads the ledger before acting, and a
build that has lost can neither report progress nor publish. Redelivered
webhooks deduplicate, an already-published revision is skipped, and a
build that stops reporting progress is replaced instead of wedging its
revision.

Both consumers boot from the published image. The Eve sandbox template
provisions from the same phases and rotates when the fingerprint changes
or a newer image is published, then fast-forwards each session to the
current `main`. Harness sessions use the snapshot as their source
instead of a stock runtime plus a clone, and fall back to cloning when
no image matches the deployment's toolchain.

The operator page gained a factory image panel showing the published
snapshot, the toolchain fingerprint, and recent builds, with a button
that rebuilds from the current `main` head.

Validated against a real `vercel/eve:latest` sandbox: every phase runs
green through verification in about two minutes, installing LLD, Cap'n
Proto 1.1.0, protoc 26.1, Zig 0.15.2, pnpm 10.28.0, Rust
nightly-2026-05-22, the workspace `node_modules`, a warm Cargo registry,
and hyperfine, cargo-bloat, and twiggy. Unit tests cover the pinned
versions against the files they mirror (including the dev container),
the phase list and generated script, the fingerprint, the ledger's
supersede rules, and webhook signature and event filtering.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Anthony Shew <35677084+anthonyshew@users.noreply.github.com>
@anthonyshew
anthonyshew requested review from a team and tknickman August 19, 2026 23:51
@vercel

vercel Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
examples-basic-web Ready Ready Preview Aug 20, 2026 1:09pm
examples-designsystem-docs Ready Ready Preview Aug 20, 2026 1:09pm
examples-gatsby-web Ready Ready Preview Aug 20, 2026 1:09pm
examples-kitchensink-blog Ready Ready Preview Aug 20, 2026 1:09pm
examples-nonmonorepo Ready Ready Preview Aug 20, 2026 1:09pm
examples-svelte-web Ready Ready Preview Aug 20, 2026 1:09pm
examples-tailwind-web Ready Ready Preview Aug 20, 2026 1:09pm
examples-vite-web Ready Ready Preview Aug 20, 2026 1:09pm
turbo-site Ready Ready Preview Aug 20, 2026 1:09pm
turborepo-factory Ready Ready Preview Aug 20, 2026 1:09pm

Comment thread apps/factory/app/factory-image.tsx Outdated
Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com>
@anthonyshew
anthonyshew merged commit 1c165b3 into main Aug 22, 2026
50 of 53 checks passed
@anthonyshew
anthonyshew deleted the factory-snapshot-workflow branch August 22, 2026 13:07
github-actions Bot added a commit that referenced this pull request Aug 25, 2026
## Release v2.10.12

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/32882264217)

### Changes

- chore: Release Turborepo 2.10.11 (#13765) (`1fb1e86`)
- feat: Refresh documentation social cards (#13764) (`510777b`)
- fix: Preserve showcase logo sizes (#13766) (`b03ec9d`)
- fix: Refine mobile homepage interactions (#13767) (`a253dce`)
- fix: Align homepage KPIs to the right (#13769) (`373fbd3`)
- fix: Prevent homepage KPI overflow (#13770) (`fc02d72`)
- chore: Update Geistdocs to 1.20.4 (#13771) (`0689ad9`)
- perf: Skip Unused Workspace Config Stats (#13746) (`5503fde`)
- chore: Update with-solid example (#13752) (`43c555d`)
- chore: Update with-tailwind example (#13762) (`c436362`)
- perf: Stream dry-run JSON output (#13760) (`8138ce0`)
- chore: Update with-ultracite example (#13772) (`37be819`)
- perf: Merge same-prefix tree-wildcard globs into one directory walk
(#13763) (`59d4901`)
- chore: Update with-svelte example (#13118) (`bebcfc8`)
- chore: Add unified agent control plane (#13757) (`695a172`)
- fix: Remove incremental task caching (#13773) (`3d69e42`)
- chore: Rename agents app to factory (#13774) (`0a01c2d`)
- chore: Update oxlint and oxfmt (#13777) (`9088245`)
- feat: Add security.txt endpoint (#13778) (`5f9260b`)
- feat: Redesign factory control plane (#13775) (`ffd06d4`)
- perf: Replace regex captures with hand-written parsers in berry
lockfile identifiers (#13776) (`00e9f66`)
- chore: Upgrade the factory eve agent to 0.39.3 (#13783) (`c4fc5c8`)
- chore: Update with-rsbuild-module-federation example (#13786)
(`45b1257`)
- perf: Index Berry lockfile resolution overrides by dependency name
(#13787) (`e89eddd`)
- chore: Update remote cache action to v1.1.0 (#13789) (`56162b8`)
- docs: Fix reference links and validation (#13784) (`0423d70`)
- feat: Add SSH command affordance for factory sandboxes (#13779)
(`0cfccdd`)
- fix: Show invalid affected task glob (#13793) (`ef1ef92`)
- perf: Skip unused repository indexing for package listings (#13792)
(`acd5ae5`)
- feat: Rebuild the factory image on every merge to main (#13781)
(`1c165b3`)
- chore: Add full-page terminal SSH sessions to Factory sandbox
inventory (#13780) (`d640c2e`)
- chore: Update with-shell-commands example (#13791) (`476e382`)
- chore: Allow SSH terminal for completed-run sandboxes in Factory
(#13782) (`067dfba`)
- fix: Decouple graceful shutdown tests from the shell commands example
(#13799) (`72fac33`)
- feat: Start ad-hoc factory work from the operator page (#13798)
(`b9d13ca`)
- chore: Update with-solid example (#13797) (`d412981`)
- fix: Rebuild factory images without custom workflows (#13801)
(`0c54a80`)
- feat: Add factory navigation (#13802) (`092ee6d`)
- refactor: Migrate Factory styles to Tailwind (#13803) (`7fe373b`)
- feat: Add durable Factory workspaces (#13804) (`ccd79d3`)
- feat: Stream Factory sandbox as terminal (#13807) (`33d8b24`)
- fix: Restore Factory workspace creation (#13812) (`fd72cca`)
- fix: Update Factory session network policy (#13814) (`c579fec`)
- fix: Improve Factory terminal line spacing (#13815) (`a88e39b`)
- fix: Install Factory publishing skill (#13817) (`7b8cb14`)
- chore: Route Factory publishing through Eve (#13816) (`e05b81c`)
- feat: Standardize Factory meta titles to Turborepo suffix (#13819)
(`fd593b5`)
- perf: Skip Factory chat verification (#13820) (`39137a6`)
- fix: Restore Factory workspaces (#13823) (`0afd4b2`)
- docs: Fix inconsistent Yarn command in basic example (#13825)
(`3dd49d1`)
- chore: Update non-monorepo example (#13808) (`d2a673f`)
- perf: Batch package detail queries (#13809) (`331183e`)
- chore: Update basic example (#13824) (`f06836d`)
- fix: Include virtual tasks in affected query (#13805) (`89a9b78`)
- fix: Add workspace approval controls (#13827) (`02dfd21`)
- fix: Prevent chat SSH command overflow (#13828) (`d9eaff5`)
- fix: Update Factory pull request branches (#13831) (`b670754`)
- chore: Add operator chat model selector (#13833) (`bb2fcb3`)
- fix: Move model selector to workspace creation (#13835) (`72805d7`)
- chore: Skip redundant Factory PR approval (#13837) (`bbe5406`)
- chore: Use geistdocs 1.23.1 (#13834) (`3787c06`)
- chore: Handle feedback on Factory pull requests (#13836) (`a76330b`)
- fix: Escape ampersands in RSS feed enclosure URLs (#13839) (`7107f26`)
- chore: Add automatic issue handling (#13840) (`e1674e4`)
- chore: Alert Slack for low-confidence issues (#13841) (`f153cda`)
- feat: Require high confidence for issue fixes (#13842) (`c97782b`)
- fix: Run pnpm directly on Windows (#13843) (`9d2b03b`)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
anthonyshew pushed a commit that referenced this pull request Aug 27, 2026
## Release v2.10.13-canary.1

> [!CAUTION]
> Versioned docs aliasing FAILED. [View
logs](https://github.com/vercel/turborepo/actions/runs/33018929115)

### Changes

- chore: Release Turborepo 2.10.11-canary.4 (#13759) (`9f94a7d`)
- feat: Expand performance agent toolbox (#13761) (`f924510`)
- docs: Redesign Turborepo homepage (#13702) (`09bf969`)
- fix: Tolerate transient input files (#13734) (`3226457`)
- chore: Release Turborepo 2.10.11 (#13765) (`1fb1e86`)
- feat: Refresh documentation social cards (#13764) (`510777b`)
- fix: Preserve showcase logo sizes (#13766) (`b03ec9d`)
- fix: Refine mobile homepage interactions (#13767) (`a253dce`)
- fix: Align homepage KPIs to the right (#13769) (`373fbd3`)
- fix: Prevent homepage KPI overflow (#13770) (`fc02d72`)
- chore: Update Geistdocs to 1.20.4 (#13771) (`0689ad9`)
- perf: Skip Unused Workspace Config Stats (#13746) (`5503fde`)
- chore: Update with-solid example (#13752) (`43c555d`)
- chore: Update with-tailwind example (#13762) (`c436362`)
- perf: Stream dry-run JSON output (#13760) (`8138ce0`)
- chore: Update with-ultracite example (#13772) (`37be819`)
- perf: Merge same-prefix tree-wildcard globs into one directory walk
(#13763) (`59d4901`)
- chore: Update with-svelte example (#13118) (`bebcfc8`)
- chore: Add unified agent control plane (#13757) (`695a172`)
- fix: Remove incremental task caching (#13773) (`3d69e42`)
- chore: Rename agents app to factory (#13774) (`0a01c2d`)
- chore: Update oxlint and oxfmt (#13777) (`9088245`)
- feat: Add security.txt endpoint (#13778) (`5f9260b`)
- feat: Redesign factory control plane (#13775) (`ffd06d4`)
- perf: Replace regex captures with hand-written parsers in berry
lockfile identifiers (#13776) (`00e9f66`)
- chore: Upgrade the factory eve agent to 0.39.3 (#13783) (`c4fc5c8`)
- chore: Update with-rsbuild-module-federation example (#13786)
(`45b1257`)
- perf: Index Berry lockfile resolution overrides by dependency name
(#13787) (`e89eddd`)
- chore: Update remote cache action to v1.1.0 (#13789) (`56162b8`)
- docs: Fix reference links and validation (#13784) (`0423d70`)
- feat: Add SSH command affordance for factory sandboxes (#13779)
(`0cfccdd`)
- fix: Show invalid affected task glob (#13793) (`ef1ef92`)
- perf: Skip unused repository indexing for package listings (#13792)
(`acd5ae5`)
- feat: Rebuild the factory image on every merge to main (#13781)
(`1c165b3`)
- chore: Add full-page terminal SSH sessions to Factory sandbox
inventory (#13780) (`d640c2e`)
- chore: Update with-shell-commands example (#13791) (`476e382`)
- chore: Allow SSH terminal for completed-run sandboxes in Factory
(#13782) (`067dfba`)
- fix: Decouple graceful shutdown tests from the shell commands example
(#13799) (`72fac33`)
- feat: Start ad-hoc factory work from the operator page (#13798)
(`b9d13ca`)
- chore: Update with-solid example (#13797) (`d412981`)
- fix: Rebuild factory images without custom workflows (#13801)
(`0c54a80`)
- feat: Add factory navigation (#13802) (`092ee6d`)
- refactor: Migrate Factory styles to Tailwind (#13803) (`7fe373b`)
- feat: Add durable Factory workspaces (#13804) (`ccd79d3`)
- feat: Stream Factory sandbox as terminal (#13807) (`33d8b24`)
- fix: Restore Factory workspace creation (#13812) (`fd72cca`)
- fix: Update Factory session network policy (#13814) (`c579fec`)
- fix: Improve Factory terminal line spacing (#13815) (`a88e39b`)
- fix: Install Factory publishing skill (#13817) (`7b8cb14`)
- chore: Route Factory publishing through Eve (#13816) (`e05b81c`)
- feat: Standardize Factory meta titles to Turborepo suffix (#13819)
(`fd593b5`)
- perf: Skip Factory chat verification (#13820) (`39137a6`)
- fix: Restore Factory workspaces (#13823) (`0afd4b2`)
- docs: Fix inconsistent Yarn command in basic example (#13825)
(`3dd49d1`)
- chore: Update non-monorepo example (#13808) (`d2a673f`)
- perf: Batch package detail queries (#13809) (`331183e`)
- chore: Update basic example (#13824) (`f06836d`)
- fix: Include virtual tasks in affected query (#13805) (`89a9b78`)
- fix: Add workspace approval controls (#13827) (`02dfd21`)
- fix: Prevent chat SSH command overflow (#13828) (`d9eaff5`)
- fix: Update Factory pull request branches (#13831) (`b670754`)
- chore: Add operator chat model selector (#13833) (`bb2fcb3`)
- fix: Move model selector to workspace creation (#13835) (`72805d7`)
- chore: Skip redundant Factory PR approval (#13837) (`bbe5406`)
- chore: Use geistdocs 1.23.1 (#13834) (`3787c06`)
- chore: Handle feedback on Factory pull requests (#13836) (`a76330b`)
- fix: Escape ampersands in RSS feed enclosure URLs (#13839) (`7107f26`)
- chore: Add automatic issue handling (#13840) (`e1674e4`)
- chore: Alert Slack for low-confidence issues (#13841) (`f153cda`)
- feat: Require high confidence for issue fixes (#13842) (`c97782b`)
- fix: Run pnpm directly on Windows (#13843) (`9d2b03b`)
- chore: Release Turborepo 2.10.12 (#13844) (`32748f5`)
- fix: Remove unsupported remote cache environment variable (#13845)
(`b4c2eed`)
- fix: Copy TUI selections locally over SSH (#13847) (`03df632`)
- feat: Use uv workspace metadata (#13848) (`fa1ca7d`)
- feat: Support Python virtual environments (#13849) (`7f66dbd`)
- fix: Scope uv lockfile affectedness (#13850) (`1e074f3`)
- test: Isolate uv prune configuration (#13851) (`9f2fd33`)
- fix: Explain disabled uv task caching (#13852) (`0f59d11`)
- fix: Explain uv identity probe failures (#13853) (`eabe73a`)
- fix: Explain uncached Cargo library builds (#13855) (`35ce2fa`)
- fix: Explain disabled Cargo task caching (#13854) (`39821f6`)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview – turborepo-factory c1788871 Deployed Aug 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant