docker: multi-stage images for most architectures, CI, and fly rewrite

[ndizazzo]

Summary

This PR adds five multi-stage Docker images (client, cpu, vulkan, cuda, rocm), a CI workflow that publishes them as native multi-arch manifests, and fixes a latent bug in fly/Dockerfile that would break any cleanroom deploy.

What you can do now

# Smallest, most portable — runs the web console
docker run --rm -p 3131:3131 -p 9337:9337 \
  -e APP_MODE=console ghcr.io/<owner>/mesh-llm:latest

# Full worker nodes with bundled llama.cpp
docker run --rm -p 9337:9337 -v ~/.models:/root/.models \
  -e APP_MODE=worker ghcr.io/<owner>/mesh-llm:cpu

# GPU workers
docker run --rm --gpus all -p 9337:9337 \
  -v ~/.models:/root/.models -e APP_MODE=worker \
  ghcr.io/<owner>/mesh-llm:cuda

docker run --rm --device=/dev/kfd --device=/dev/dri -p 9337:9337 \
  -v ~/.models:/root/.models -e APP_MODE=worker \
  ghcr.io/<owner>/mesh-llm:rocm

Or build locally with the new just recipes:

just docker-build-client
just docker-build-cpu
just docker-build-vulkan
just docker-build-cuda   # override arch: just docker-build-cuda "mesh-llm:cuda" "75;80"
just docker-build-rocm

Images published

Tag	Variant	Arch	Size
:latest, :client	client-only (no llama.cpp, no GPU)	amd64 + arm64	~54 MB
:cpu	full node, llama.cpp CPU	amd64 + arm64	~58 MB
:vulkan	full node, Vulkan backend	amd64 + arm64	~75 MB
:cuda	full node, CUDA 12.8.0	amd64	~2.7 GB
:rocm	full node, ROCm 7.0	amd64	~1.4 GB

All images ship a shared entrypoint with three modes selected via APP_MODE:

• console (default) — client node, API on :9337, web console on :3131
• worker — full node with bundled llama binaries at /usr/local/lib/mesh-llm/bin/
• (empty) — passthrough, args go straight to mesh-llm

The UI is embedded in the binary (include_dir!) and always co-deployed with the runtime — there is no UI-less mode.

Fly.io deploys now actually work from clean state

fly/Dockerfile previously used a single FROM rust:latest stage with no UI build step. It relied on mesh-llm/ui/dist/ already existing in the build context, so any deploy from a fresh checkout would hit the include_dir! compile-time macro and fail.

This PR rewrites it as a 4-stage multi-stage build structurally identical to docker/Dockerfile.client, with node:24-alpine building the UI and debian:bookworm-slim for the runtime. fly.toml is untouched — the runtime contract (EXPOSE 3131 9337, ENV APP_MODE=console, ENTRYPOINT ["/entrypoint.sh"]) is preserved, so fly deploy --config fly/console/fly.toml --dockerfile fly/Dockerfile works unchanged. Cleanroom worktree builds are part of the test evidence.

Architecture

Multi-stage pattern (all Dockerfiles). Four or five stages: ui-builder (node:24-alpine) → rust-deps (cargo-chef 0.1.68 with full workspace manifests + mesh-llm/proto/) → rust-builder (overwrites stub ui/dist with real build before cargo build --release --locked) → optional llama-builder-<flavor> (clones michaelneale/llama.cpp@upstream-latest, shared cmake flags GGML_RPC=ON BUILD_SHARED_LIBS=OFF LLAMA_OPENSSL=OFF CMAKE_BUILD_TYPE=Release) → runtime.

Llama binaries land at /usr/local/lib/mesh-llm/bin/{rpc-server,llama-server}-<flavor> with an unsuffixed llama-moe-split shared across flavors. The llama.cpp commit SHA is baked into /usr/local/share/mesh-llm/llama-revision.txt and an OCI label.

CUDA correctness. docker/Dockerfile.cuda ships -DGGML_CUDA_FA_ALL_QUANTS=ON — this is a correctness requirement, not a perf tuning knob. Without it, asymmetric K/V quantization paths hit BEST_FATTN_KERNEL_NONE and crash rpc-server (see ggml-org/llama.cpp#20866). The CI workflow grep-verifies GGML_CUDA_FA_ALL_QUANTS:BOOL=ON appears in the CMake cache on every CUDA build, so the flag cannot silently drop.

Spec compliance harness. scripts/verify-docker-spec.sh enforces 13 invariants across all Dockerfiles and the workflow (ui-builder before cargo build, libdbus-1-dev for keyring sync-secret-service, no just bundle, no protobuf-compiler apt package since protoc is vendored via protoc_bin_vendored, no QEMU, flavored binary naming, runtime libs ca-certificates libgomp1 libdbus-1-3, etc.). Current state: 52 PASSED, 0 FAILED. docker/SPEC_COMPLIANCE.md tracks each item with file:line references and a verification log.

CI. .github/workflows/docker.yml defines 11 jobs: client/cpu/vulkan × (amd64, arm64, merge) + cuda + rocm amd64-only. ARM64 uses native ubuntu-24.04-arm runners — no QEMU anywhere. Merge jobs stitch per-arch digests with docker buildx imagetools create. GHA cache scopes are split so a Rust-only change invalidates only the llama layer of one variant, and CUDA/ROCm caches are isolated from debian-based variants (different base images). release.yml is untouched — no coupling in either direction.

Validation

• bash scripts/verify-docker-spec.sh → 52 PASSED / 0 FAILED / 0 SKIPPED
• All 5 images built and smoke-tested locally: --version returns mesh-llm 0.58.0, flavored binaries present at expected paths, ldd reports zero unresolved libs on the client image, all images under their size budgets (250 MB / 600 MB / 900 MB / 4.5 GB / 12 GB respectively).
• fly/Dockerfile cleanroom build verified via git worktree add with ui/dist pre-deleted.
• just docker-build-* recipes dry-runned with arch overrides.
• Entrypoint tested in all three modes.

Files changed

.dockerignore                    +14
.github/workflows/docker.yml     new (590 lines)
Justfile                         +34 (7 new recipes, append-only)
docker/Dockerfile.client         new
docker/Dockerfile.cpu            new
docker/Dockerfile.cuda           new
docker/Dockerfile.rocm           new
docker/Dockerfile.vulkan         new
docker/entrypoint.sh             new
docker/SPEC_COMPLIANCE.md        new
fly/Dockerfile                   rewritten (+66/-33)
scripts/verify-docker-spec.sh    new

No changes to mesh-llm/src/, Cargo.toml, Cargo.lock, release.yml, or fly.toml.

Notes for reviewers

• Stages 1–3 of all Dockerfiles are intentionally near-identical copies rather than shared via a common base image — Docker doesn't support cross-file stage inheritance, and keeping them self-contained avoids drift between the fly deploy and the published :client image. docker/SPEC_COMPLIANCE.md calls this out.
• cargo-chef is pinned to 0.1.68 --locked and the cook step now also uses --locked for reproducibility.
• The Vulkan image installs Khronos Vulkan-Headers v1.3.296 on top of Debian Bookworm's libvulkan-dev (1.3.239) because current llama.cpp@upstream-latest needs VK_EXT_layer_settings from 1.3.261+. The runtime loader from Debian is kept; only headers are upgraded.
• ROCm runtime uses the plain rocm/dev-ubuntu-24.04:7.0 base (not -complete) to keep the image reasonable — -complete is ~20 GB.
• Images run as root to match the existing fly/Dockerfile convention; K3S/pod users can override via securityContext.runAsNonRoot.

Closes #115

[Copilot]

Pull request overview

This PR introduces a container build/publish pipeline for mesh-llm by adding multi-stage Docker images for multiple runtime variants (client, CPU, Vulkan, CUDA, ROCm), plus a GitHub Actions workflow to build/push them (including multi-arch manifests where applicable). It also rewrites fly/Dockerfile to include a UI build stage so Fly deploys work from a clean checkout.

Changes:

• Added multi-stage Dockerfiles for client/CPU/Vulkan/CUDA/ROCm variants and a shared docker/entrypoint.sh.
• Added .github/workflows/docker.yml to build/push per-arch images and merge multi-arch manifests for select variants.
• Added a spec-compliance harness (scripts/verify-docker-spec.sh) and human tracker (docker/SPEC_COMPLIANCE.md); updated Justfile with Docker build/run recipes.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
.dockerignore	Ignores Node/UI artifacts and local/editor state to keep Docker contexts clean
.github/workflows/docker.yml	New CI workflow to build/push Docker images and create multi-arch manifests
Justfile	Adds just docker-build-* and docker-run-* helper recipes
docker/Dockerfile.client	Multi-stage client-only image (UI build + Rust build)
docker/Dockerfile.cpu	Multi-stage CPU image including llama.cpp CPU build
docker/Dockerfile.vulkan	Multi-stage Vulkan image including llama.cpp Vulkan build
docker/Dockerfile.cuda	Multi-stage CUDA image including llama.cpp CUDA build
docker/Dockerfile.rocm	Multi-stage ROCm image including llama.cpp ROCm build
docker/entrypoint.sh	Shared runtime entrypoint selecting behavior via APP_MODE
docker/SPEC_COMPLIANCE.md	Human-readable checklist mirroring the verifier script
fly/Dockerfile	Rewritten multi-stage Fly console image including UI build stage
scripts/verify-docker-spec.sh	New script enforcing Docker/spec invariants across files

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

[i386]

LGTM - I would just double check some of the copilot comments before merging.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.