Skip to content

feat(workspace-server): GHCR digest watcher closes runtime CD chain - #2114

Merged
HongmingWang-Rabbit merged 1 commit into
stagingfrom
feat/image-auto-refresh
Apr 26, 2026
Merged

feat(workspace-server): GHCR digest watcher closes runtime CD chain#2114
HongmingWang-Rabbit merged 1 commit into
stagingfrom
feat/image-auto-refresh

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in watcher in workspace-server that polls GHCR every 5 minutes for digest changes on each `workspace-template-*:latest` tag. When a digest moves, it invokes the same refresh logic the existing `POST /admin/workspace-images/refresh` endpoint exposes.

With this, the runtime CD chain is fully hands-off end-to-end:

```
merge runtime PR → auto-tag → publish-runtime (OIDC → PyPI)
→ cascade (repository_dispatch → 8 templates)
→ template publish-image.yml (GHCR push)
→ ★ this watcher ★ (GHCR digest polled, change detected, host pulls + recreates)
→ containers running new code
```

No operator step between any of those arrows.

Opt-in via env

`IMAGE_AUTO_REFRESH=true` on the platform process. SaaS deploys whose pipeline already pulls every release should leave it disabled (would be redundant work). Self-hosters get true zero-touch.

What's in the PR

File Why
`workspace-server/internal/imagewatch/watch.go` (new) The watcher: GHCR token + manifest HEAD, in-memory digest tracking, supervised goroutine entry point
`workspace-server/internal/imagewatch/watch_test.go` (new) Covers all tick() branches: seed-on-first, no-refresh-on-steady-state, refresh-on-change, rollback-on-error, fetch-error-skip-this-runtime-only
`workspace-server/internal/handlers/admin_workspace_images.go` (refactor) Extracts `WorkspaceImageService` + `Refresh(ctx, runtimes, recreate) (RefreshResult, error)` so the watcher and HTTP handler share one code path. HTTP handler is now a thin wrapper; behavior preserved (same JSON, same 500-on-list-failure, same soft-fail).
`workspace-server/cmd/server/main.go` Wires `supervised.RunWithRecover` for the watcher when `IMAGE_AUTO_REFRESH=true` and a Docker provisioner exists
`docs/workspace-runtime-package.md` New "Fully hands-off (opt-in)" subsection under Step 5

Design choices worth flagging

  • In-memory digest tracking, not persisted. First observation per runtime is seed-only — no spurious refresh on every restart. Trade-off: a digest change during downtime is missed; operator can hit the existing endpoint once after deploy. Persisting felt over-engineered for a 5-min-cadence observer.
  • Rollback-on-error. If `Refresh` returns an error, the seen-digest rolls back. Without this, a transient Docker glitch would convince the watcher the work was done.
  • Per-runtime fetch error doesn't block others. One template's brief 500 doesn't pause the rest.
  • `digestFetcher` injection seam. `tick(ctx, fetch)` takes the digest function as a parameter. Production passes `w.remoteDigest`; tests pass deterministic fakes. Avoids 100+ lines of httptest scaffolding for a 60-line watcher.

Verified live

Probed GHCR's `/token` + manifest HEAD against `workspace-template-claude-code` (the same calls the watcher makes):

```
HTTP/2 200
content-type: application/vnd.oci.image.index.v1+json
docker-content-digest: sha256:a200d7446a3f49ff6b953147849cbba671889ccf22eb0ed9f58db2a6e1828838
```

Test plan

  • `go build ./...` clean
  • `go vet ./...` clean
  • `go test ./internal/imagewatch/ ./internal/handlers/` green (5 watcher tests, 4 existing handler tests)
  • Set `IMAGE_AUTO_REFRESH=true` on a staging deploy, publish a dev runtime version, observe `image-auto-refresh: digest moved … → …, refreshing` in platform logs within 5 minutes
  • Confirm matching ws-* containers get force-removed and re-provision on next interaction

What this does not do (deferred to separate work)

🤖 Generated with Claude Code

Adds an opt-in goroutine that polls GHCR every 5 minutes for digest
changes on each workspace-template-*:latest tag and invokes the same
refresh logic /admin/workspace-images/refresh exposes. With this, the
chain from "merge runtime PR" to "containers running new code" is fully
hands-off — no operator step between auto-tag → publish-runtime →
cascade → template image rebuild → host pull + recreate.

Opt-in via IMAGE_AUTO_REFRESH=true. SaaS deploys whose pipeline already
pulls every release should leave it off (would be redundant work);
self-hosters get true zero-touch.

Why a refactor of admin_workspace_images.go is in this PR:
The HTTP handler held all the refresh logic inline. To share it with
the new watcher without HTTP loopback, extracted WorkspaceImageService
with a Refresh(ctx, runtimes, recreate) (RefreshResult, error) shape.
HTTP handler is now a thin wrapper; behavior is preserved (same JSON
response, same 500-on-list-failure, same per-runtime soft-fail).

Watcher design notes:
- Last-observed digest tracked in memory (not persisted). On boot the
  first observation per runtime is seed-only — no spurious refresh
  fires on every restart.
- On Refresh error, the seen digest rolls back so the next tick retries.
  Without this rollback a transient Docker glitch would convince the
  watcher the work was done.
- Per-runtime fetch errors don't block other runtimes (one template's
  brief 500 doesn't pause the others).
- digestFetcher injection seam in tick() lets unit tests cover all
  bookkeeping branches without standing up an httptest GHCR server.

Verified live: probed GHCR's /token + manifest HEAD against
workspace-template-claude-code; got HTTP 200 + a real
Docker-Content-Digest. Same calls the watcher makes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit
HongmingWang-Rabbit merged commit 9375e3d into staging Apr 26, 2026
14 checks passed
HongmingWang-Rabbit added a commit that referenced this pull request Apr 26, 2026
Picks up the GHCR digest watcher added in PR #2114 with no operator
action: just `docker compose up` and the platform self-heals to the
latest workspace-template image within 5 minutes of publish.

Default ON for local dev because that's where the runtime → workspace
iteration loop is tightest. .env.example documents the override knob
for the rare "running a long test that shouldn't be disturbed by a
publish" case.

Co-authored-by: Hongming Wang <hongmingwangalt@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
HongmingWang-Rabbit pushed a commit that referenced this pull request Apr 26, 2026
Two paper cuts the fix addresses:

1. nuke-and-rebuild.sh wipes the compose stack but never re-populates
   workspace-configs-templates/, org-templates/, or plugins/. Those dirs
   are .gitignored — the curated set lives in manifest.json as external
   repos cloned via clone-manifest.sh (idempotent). Without that step,
   a fresh checkout or a post-deletion run leaves the dirs empty, which
   silently hides the entire template palette in Canvas + falls back to
   bare default workspace provisioning. Symptom: "Deploy your first
   agent" shows zero templates.

2. The existing ws-* container reap was already in the script (good),
   but it only fires when this script runs. Folks running `docker compose
   down -v` directly leave orphan ws-* containers behind. Documented
   that explicitly in the script comment so future readers understand
   why those lines are critical.

The fix is just `bash clone-manifest.sh` added to the script. clone-
manifest.sh is idempotent — populated dirs short-circuit, so a re-nuke
on a healthy machine pays only a few stat calls.

scripts/test-nuke-and-rebuild.sh exercises the canonical workflow end-
to-end:
  - plants a fake orphan ws-* container, then asserts it gets reaped
  - renames the manifest dirs to simulate a fresh checkout, then
    asserts they get repopulated
  - waits for /health and asserts the platform sees the same template
    count on disk as via /configs in the container (catches bind-mount
    drift)
  - asserts the image-auto-refresh watcher (PR #2114) starts, since
    that's load-bearing for the CD chain users now rely on

The test pre-flights port 5432/6379/8080 and exits 0 with a SKIP
message if a non-target compose project is holding them — common when
parallel monorepo checkouts coexist on one Docker daemon.

scripts/ is intentionally outside CI shellcheck per ci.yml comment, but
both files pass `shellcheck --severity=warning` anyway.

Defers but does not solve the runtime root-cause for orphan ws-* after
plain `docker compose down -v`: the orphan-sweeper in the platform only
reaps containers whose workspace row says status='removed', so a wiped
DB → no row → sweeper ignores them. Proper fix needs container labels
keyed to a per-platform-instance UUID so the sweeper can confidently
reap "containers I provisioned that aren't in my DB anymore" without
nuking a sibling platform's containers on a shared daemon. Tracked as
task #109's follow-up; out of scope for this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@molecule-ai
molecule-ai Bot deleted the feat/image-auto-refresh branch May 20, 2026 06:21
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…overage (#1312, clean extract)' (#2114) from test/org-scope-abilities-coverage-clean into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant