task: unify release hosts, native runtimes, and product bundles

.agents/skills/manage-ci/SKILL.md

@@ -220,6 +220,24 @@ update the skill resources in the same change.
 
 ## Caches, artifacts, and smoke tests
 
+- Treat release production as a three-layer graph: one backend-neutral host per
+  OS/architecture, one native-runtime artifact per
+  OS/architecture/backend/backend-version, and product composition from those
+  two immutable inputs. A backend alias may select a runtime but must never
+  compile a distinct host.
+- Host artifacts must include and pass the dynamic-import policy report.
+  Runtime artifacts must include their manifest, file checksums, runtime ABI,
+  MeshLLM version, platform, and backend compatibility. Product artifacts must
+  record the exact host and runtime digests and preserve the
+  `mesh-bundle/native-runtimes/<runtime-id>` layout.
+- Keep host, runtime, and product matrices separate in release workflows.
+  Product jobs download producer artifacts and verify compatibility and digest
+  metadata before composition. Never satisfy a missing producer by rebuilding
+  either layer in a consumer job.
+- No-driver smoke is mandatory for every product alias, native package, and OCI
+  image. CUDA/ROCm/Vulkan device absence is not a skip condition for
+  `--version`, runtime discovery/listing, or client startup. Hardware-qualified
+  serving tests are additional coverage.
 - Namespace cache keys and include every compatibility boundary that can make
   reuse unsafe: OS, architecture, backend/toolchain, relevant lockfiles,
   `.github/cache-version.txt`, and build inputs. Do not broaden restore keys

.agents/skills/manage-ci/references/current-inventory.md

@@ -31,6 +31,20 @@ the commands at the end before operational changes.
 | `reset-caches.yml` | Confirmed dispatch | Destructive repository cache reset |
 | `stale-prs.yml` | Schedule, dispatch | PR warning/closure maintenance |
 
+`release.yml` is migrating to the contract-v2 three-layer artifact graph:
+
+- host matrix: one dynamic, backend-neutral executable and import report per
+  supported OS/architecture;
+- native-runtime matrix: one manifested/checksummed runtime per supported
+  OS/architecture/backend/backend-version;
+- product matrix: digest-verified composition of a host plus one runtime while
+  retaining existing backend-flavored public archive names as aliases.
+
+Release and packaging consumers must not rebuild either input. Portable bundles
+place runtimes at `mesh-bundle/native-runtimes/<runtime-id>`; Debian/Arch
+packages use `/usr/local/lib/mesh-llm/<version>/native-runtimes`; Homebrew uses
+formula-owned `libexec/native-runtimes`.
+
 Local actions:
 
 - `.github/actions/compute-changes` owns path, crate, backend, SDK, UI, website,

.github/workflows/release.yml

@@ -832,7 +832,7 @@ jobs:
         uses: actions/cache@v5
         with:
           path: ${{ env.LLAMA_STAGE_BUILD_DIR }}
-          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-cuda${{ matrix.cuda_major }}-${{ matrix.cuda_version }}-static-${{ matrix.cuda_architectures_cache }}-${{ hashFiles('scripts/build-linux.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
+          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-cuda${{ matrix.cuda_major }}-${{ matrix.cuda_version }}-dynamic-${{ matrix.cuda_architectures_cache }}-${{ hashFiles('scripts/package-native-runtime.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
       - name: Prepare dispatched release version
         if: github.event_name == 'workflow_dispatch'
         env:
@@ -886,7 +886,7 @@ jobs:
         uses: actions/cache@v5
         with:
           path: ${{ env.LLAMA_STAGE_BUILD_DIR }}
-          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-rocm7.0-static-gfx90a_gfx942_gfx1100_gfx1101_gfx1102_gfx1103_gfx1151_gfx1200_gfx1201-${{ hashFiles('scripts/build-linux.sh', 'scripts/build-linux-rocm.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
+          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-rocm7.0-dynamic-gfx90a_gfx942_gfx1100_gfx1101_gfx1102_gfx1103_gfx1151_gfx1200_gfx1201-${{ hashFiles('scripts/package-native-runtime.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
       - name: Prepare dispatched release version
         if: github.event_name == 'workflow_dispatch'
         env:
@@ -937,7 +937,7 @@ jobs:
         uses: actions/cache@v5
         with:
           path: ${{ env.LLAMA_STAGE_BUILD_DIR }}
-          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-vulkan-static-${{ hashFiles('scripts/build-linux.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
+          key: ${{ env.CACHE_NAMESPACE }}-release-llama-linux-x86_64-vulkan-dynamic-${{ hashFiles('scripts/package-native-runtime.sh', 'scripts/build-llama.sh', 'scripts/prepare-llama.sh', 'Justfile', 'third_party/llama.cpp/upstream.txt', 'third_party/llama.cpp/patches/**', '.github/cache-version.txt') }}
       - name: Prepare dispatched release version
         if: github.event_name == 'workflow_dispatch'
         env:

AGENTS.md

@@ -86,13 +86,12 @@ just ui-clean      # nuke node_modules + dist (fixes stale npm state)
   or surface bugs that release builds don't.
 - `just release-build` → produces `./target/release/mesh-llm`. Use this for any
   serious testing, deploying to test machines, bundling, or releases. Release
-  builds default to `MESH_LLM_DYNAMIC_NATIVE_RUNTIME=1`, so the binary loads a
-  compatible installed native runtime at startup instead of embedding the
-  branch-local llama.cpp ABI libraries. When validating branch-local Skippy ABI,
-  llama.cpp patch, MAS hidden-state, or native tensor changes, either use
-  `just build` for the normal static local dev loop or run
-  `MESH_LLM_DYNAMIC_NATIVE_RUNTIME=0 just release-build` before release-mode
-  behavior/performance testing.
+  builds always produce one backend-neutral host plus a packageable native
+  runtime. When validating branch-local Skippy ABI, llama.cpp patches, MAS
+  hidden-state, or native tensor changes, use `just release-host-build` and
+  `just release-runtime-build <backend>`, then point the host at
+  `dist/native-runtimes` with `MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR`. Static
+  backend linkage is not a release or packaging lane.
 - `./target/release/mesh-llm` may exist from a *previous* `just release-build`
   or `just build-dev` invocation even after you run only `just build` — its
   presence is **not** evidence that your latest code is in it. When in doubt,
@@ -108,6 +107,23 @@ whether you need the default dynamic release packaging path or an embedded
 branch-local native ABI; do not test new ABI symbols against downloaded release
 native runtimes.
 
+Release artifacts follow one three-layer graph:
+
+| Layer | Command | Output |
+|---|---|---|
+| Neutral host | `just release-host-build` | `target/release/mesh-llm` plus an import-policy report during packaging |
+| Native runtime | `just release-runtime-build <backend>` | `dist/native-runtimes/<runtime-id>/` plus archive/checksum |
+| Product | `just release-bundle vX.Y.Z <output>` | `mesh-bundle/` containing the host, one runtime, and product/host-import manifests |
+
+The host dependency policy is enforced by
+`scripts/verify-host-dependencies.py`. Release, installer, SDK, native-package,
+and image lanes must not bypass it or copy backend libraries beside the host.
+For an isolated local runtime test, set
+`MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes"` and a fresh
+`MESH_LLM_NATIVE_RUNTIME_CACHE_DIR`. Discovery never searches the current
+working directory. Do not reintroduce an external `llama-server` or
+`rpc-server` lane.
+
 ### npm "Exit handler never called" error
 
 If `just build` fails on the UI step with `npm error Exit handler never called!`, run:

CONTRIBUTING.md

@@ -19,90 +19,61 @@ This file covers local build and development workflows for this repository.
 
 **Linux Vulkan**: Vulkan is supported when the Vulkan development files and `glslc` are installed. On Ubuntu/Debian, install `libvulkan-dev glslc`. On Arch Linux, install `vulkan-headers shaderc`.
 
-**Windows**: `just build` auto-detects `cuda`, `hip`/`rocm`, `vulkan`, or `cpu`. You can override with `just build backend=cuda` (or `rocm`, `vulkan`, `cpu`). Metal is not supported on Windows.
+**Windows**: native runtime builds support `cuda`, `hip`/`rocm`, `vulkan`, or
+`cpu`. Metal is not supported on Windows.
 
 ## Build from source
 
-Build everything (patched llama.cpp, mesh binary, and UI production build):
+Build the normal debug development binary, patched llama.cpp ABI, and UI:
 
 ```bash
 just build
 ```
 
-`just build` builds the mesh binary in release mode. For day-to-day iteration
-where the final release link is the slow step, use the debug-profile shortcut:
+Release and packaging work use a backend-neutral host plus a separate native
+runtime. Build the host once:
 
 ```bash
-just build-dev
+just release-host-build
 ```
 
-You can also keep the normal recipe shape and select the profile explicitly:
+Then build the backend runtime you are changing:
 
 ```bash
-MESH_LLM_BUILD_PROFILE=dev just build
+just release-runtime-build cpu
+just release-runtime-build metal
+just release-runtime-build cuda
+just release-runtime-build rocm
+just release-runtime-build vulkan
 ```
 
-On Linux, `just build` auto-detects CUDA vs ROCm vs Vulkan. For NVIDIA, make sure `nvcc` is in your `PATH` first:
+Backend toolchains must be available for the corresponding runtime build. For
+NVIDIA on Linux, put `nvcc` on `PATH`; runtime packaging detects the selected
+CUDA major and architecture from the toolchain/environment.
 
 ```bash
-# Arch Linux
-PATH=/opt/cuda/bin:$PATH just build
-
-# Ubuntu/Debian
-PATH=/usr/local/cuda/bin:$PATH just build
-```
-
-For NVIDIA builds, the script auto-detects your GPU's CUDA architecture. To override:
-
-```bash
-just build cuda_arch=90   # e.g. H100
-```
-
-For AMD ROCm builds, you can force the backend explicitly:
-
-```bash
-just build backend=rocm
+PATH=/opt/cuda/bin:$PATH just release-runtime-build cuda
+# or
+PATH=/usr/local/cuda/bin:$PATH just release-runtime-build cuda
 ```
 
-To override the AMD GPU target list:
+Exercise the exact release discovery boundary with an isolated cache:
 
 ```bash
-just build backend=rocm rocm_arch="gfx90a;gfx942;gfx1100"
-```
-
-For Vulkan builds, force the backend explicitly:
-
-```bash
-just build backend=vulkan
-```
-
-For CPU-only builds (no GPU acceleration):
-
-```bash
-just build backend=cpu
-```
-
-On Windows, you can override the detected backend if needed:
-
-```powershell
-just build backend=vulkan
-just build backend=cpu
-just build backend=cuda cuda_arch=90
-```
-
-Windows release bundles use dedicated Windows release recipes:
-
-```powershell
-just release-build-cuda-windows
-just release-bundle-cuda-windows v0.X.0
+runtime_cache="$(mktemp -d)"
+MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
+MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$runtime_cache" \
+  ./target/release/mesh-llm runtime list
 ```
 
-GitHub Actions uses Blacksmith Windows 2025 runners for compile-only Windows CI and release bundle validation.
+The resolver validates version, Skippy ABI, OS, architecture, and backend. It
+does not search the current working directory or copy a matching bundled
+runtime into the user cache.
 
-Create a portable bundle:
+Create a portable product bundle after building both layers:
 
 ```bash
-just bundle
+just release-bundle v0.X.0 dist
 ```
 
 ## UI development workflow

Justfile

@@ -143,21 +143,34 @@ build-runtime backend="" cuda_arch="" rocm_arch="":
 release version *ARGS:
     @scripts/release.sh "{{ version }}" {{ ARGS }}
 
-# Release builds default to dynamic native runtimes. Set
-# MESH_LLM_DYNAMIC_NATIVE_RUNTIME=0 when validating a release binary with
-# branch-local llama.cpp ABI changes embedded.
-release-build:
+# Build the backend-neutral release host once for the current platform.
+release-host-build:
     @scripts/build-release.sh
 
+# Build one packageable native runtime for the current platform.
+release-runtime-build backend="" target="":
+    #!/usr/bin/env bash
+    set -euo pipefail
+    selected_backend="{{ backend }}"
+    if [[ -z "$selected_backend" ]]; then
+        if [[ "$(uname -s)" == Darwin ]]; then selected_backend=metal; else selected_backend=cpu; fi
+    fi
+    target_args=()
+    if [[ -n "{{ target }}" ]]; then target_args+=(--target "{{ target }}"); fi
+    scripts/package-native-runtime.sh --build --backend "$selected_backend" "${target_args[@]}"
+
+# Build the backend-neutral host and the default runtime for this platform.
+release-build: release-host-build release-runtime-build
+
 # Build a Linux aarch64 CPU release artifact on a native aarch64 runner.
-release-build-aarch64:
-    @scripts/build-release.sh
+release-build-aarch64: release-host-build
+    @scripts/package-native-runtime.sh --build --backend cpu --target aarch64-unknown-linux-gnu
 
 # Build a Linux aarch64 CUDA release artifact (Jetson/Orin).
 # SM arches selected by MESH_CUDA_VERSION env (set by CI matrix).
-release-build-aarch64-cuda:
-    @MESH_LLM_BUILD_PROFILE=release MESH_RELEASE_ARCH=aarch64 scripts/build-linux.sh --backend cuda \
-      --cuda-arch "$(if [[ "${MESH_CUDA_VERSION:-}" == 13.* ]]; then echo '75;80;86;87;89;90;110'; else echo '75;80;86;87;89;90'; fi)"
+release-build-aarch64-cuda: release-host-build
+    @LLAMA_STAGE_CUDA_ARCHITECTURES="$(if [[ "${MESH_CUDA_VERSION:-}" == 13.* ]]; then echo '75;80;86;87;89;90;110'; else echo '75;80;86;87;89;90'; fi)" \
+      scripts/package-native-runtime.sh --build --backend cuda --target aarch64-unknown-linux-gnu
 
 # Prepare the pinned llama.cpp checkout and apply the Mesh-LLM ABI patch queue.
 llama-prepare:
@@ -176,23 +189,24 @@ release-build-windows:
 
 # Build a Linux CUDA release artifact.
 # SM arches selected by MESH_CUDA_VERSION env (set by CI matrix).
-release-build-cuda:
-    @MESH_LLM_BUILD_PROFILE=release scripts/build-linux.sh --backend cuda \
-      --cuda-arch "$(if [[ "${MESH_CUDA_VERSION:-}" == 13.* ]]; then echo '75;80;86;87;89;90;100;103;120;121'; else echo '75;80;86;87;89;90'; fi)"
+release-build-cuda: release-host-build
+    @LLAMA_STAGE_CUDA_ARCHITECTURES="$(if [[ "${MESH_CUDA_VERSION:-}" == 13.* ]]; then echo '75;80;86;87;89;90;100;103;120;121'; else echo '75;80;86;87;89;90'; fi)" \
+      scripts/package-native-runtime.sh --build --backend cuda --target x86_64-unknown-linux-gnu
 
 release-build-cuda-windows cuda_arch="75;80;86;87;89;90":
     @powershell -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Backend cuda -CudaArch "{{cuda_arch}}" -BuildProfile release
 
 # Build a Linux ROCm ABI release artifact with an explicit architecture list.
-release-build-rocm rocm_arch="gfx90a;gfx942;gfx1100;gfx1101;gfx1102;gfx1103;gfx1151;gfx1200;gfx1201":
-    @MESH_LLM_BUILD_PROFILE=release scripts/build-linux-rocm.sh "{{ rocm_arch }}"
+release-build-rocm rocm_arch="gfx90a;gfx942;gfx1100;gfx1101;gfx1102;gfx1103;gfx1151;gfx1200;gfx1201": release-host-build
+    @LLAMA_STAGE_AMDGPU_TARGETS="{{ rocm_arch }}" \
+      scripts/package-native-runtime.sh --build --backend rocm --target x86_64-unknown-linux-gnu
 
 release-build-rocm-windows rocm_arch="gfx90a;gfx942;gfx1100;gfx1101;gfx1102;gfx1103;gfx1151;gfx1200;gfx1201":
     @powershell -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Backend rocm -RocmArch "{{rocm_arch}}" -BuildProfile release
 
 # Build a Linux Vulkan ABI release artifact.
-release-build-vulkan:
-    @MESH_LLM_BUILD_PROFILE=release scripts/build-linux.sh --backend vulkan
+release-build-vulkan: release-host-build
+    @scripts/package-native-runtime.sh --build --backend vulkan --target x86_64-unknown-linux-gnu
 
 release-build-vulkan-windows:
     @powershell -NoProfile -ExecutionPolicy Bypass -File scripts/build-windows.ps1 -Backend vulkan -BuildProfile release

README.md

@@ -164,10 +164,13 @@ llama.cpp parity queue.
 Tagged releases publish macOS bundles plus Linux CPU, Linux ARM64 CPU, Linux
 ARM64 CUDA, Linux CUDA, Linux CUDA Blackwell, Linux ROCm, Linux Vulkan, Windows
 CPU, Windows CUDA, Windows ROCm, and Windows Vulkan bundles. Metal is
-macOS-only. The Linux ARM64 CPU artifact is
+macOS-only. Every flavor is composed from the same backend-neutral host for its
+OS/architecture plus one versioned native runtime. The Linux ARM64 CPU artifact is
 `mesh-llm-aarch64-unknown-linux-gnu.tar.gz`; the Linux ARM64 CUDA artifact is
 `mesh-llm-aarch64-unknown-linux-gnu-cuda.tar.gz`. In install and release
-contexts, `arm64` and `aarch64` mean the same 64-bit ARM target.
+contexts, `arm64` and `aarch64` mean the same 64-bit ARM target. Portable
+archives work offline: the host discovers the adjacent
+`native-runtimes/<runtime-id>` tree before consulting the user cache.
 
 Build from source with `just`:
 
@@ -177,9 +180,21 @@ cd mesh-llm
 just build
 ```
 
-Source builds require `just`, `cmake`, Rust, and Node.js 24 + npm. CUDA builds
-need `nvcc`, ROCm builds need ROCm/HIP, and Vulkan builds need Vulkan dev files
-plus `glslc`.
+Source builds require `just`, `cmake`, Rust, and Node.js 24 + npm. To exercise
+the release boundary locally, build the neutral host and one runtime:
+
+```bash
+just release-host-build
+just release-runtime-build metal # or cpu, cuda, rocm, vulkan
+MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
+MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$(mktemp -d)" \
+  ./target/release/mesh-llm runtime list
+```
+
+CUDA runtimes need `nvcc`, ROCm runtimes need ROCm/HIP, and Vulkan runtimes need
+Vulkan development files plus `glslc`. See
+[docs/design/NATIVE_RUNTIMES.md](docs/design/NATIVE_RUNTIMES.md) for the
+manifest, discovery, and compatibility contract.
 
 The shipped `mesh-llm` executable uses embedded release attestation for
 provenance and admission hardening only. It does not apply to SDK, XCFramework,