Make native package releases immutable and complete

.github/workflows/images-precheck.yml

@@ -40,7 +40,7 @@ jobs:
             --test-coverage-lines=100 --test-coverage-branches=100 --test-coverage-functions=100 \
             tests/image-matrix.test.ts
           node --experimental-strip-types --test \
-            tests/client-readiness-smoke.test.ts tests/homebrew-release.test.ts tests/node-sdk-package.test.ts tests/node-sdk-runtime-smoke.test.ts tests/product-contract.test.ts tests/upstream-archive.test.ts tests/workflow-provenance.test.ts
+            tests/client-readiness-smoke.test.ts tests/homebrew-release.test.ts tests/node-sdk-package.test.ts tests/node-sdk-runtime-smoke.test.ts tests/product-contract.test.ts tests/release-evidence.test.ts tests/sbom-subject.test.ts tests/upstream-archive.test.ts tests/workflow-provenance.test.ts
       - name: Lint shell scripts
         run: |
           sudo apt-get update

TODO.md

@@ -1,5 +1,17 @@
 # Production Readiness TODO
 
+- [x] Make native release evidence exact and publication immutable.
+  Final result: every package row preserves a uniquely named BuildKit statement,
+  scans only the exact package file into SPDX, verifies the package basename and
+  SHA-256 subject, and participates in one deterministic 11-subject aggregate
+  provenance statement during full dry runs. Publication rejects filters and
+  any existing release drift; an exactly identical release is a no-op, while a
+  new release is created once without asset clobbering.
+  QA: focused SBOM/release-evidence fixtures cover both native formats and
+  reject generic, stale, duplicate, incomplete, or mismatched evidence; the
+  full TypeScript suite, matrix coverage, shellcheck, actionlint, Dockerfile
+  checks, workflow policy scans, and `git diff --check` pass.
+
 - [x] Harden composed-runtime certification teardown and dependency inspection.
   QA: shell syntax accepts `docker/qa-runtime-image.sh`; focused client-readiness
   and Node SDK smoke tests prove fail-closed captured `ldd` output, SIGTERM-first
@@ -108,8 +120,11 @@ strategy.
   - Final result: binary and package jobs publish `SHA256SUMS`, compatibility
     `.sha256` files, SPDX JSON SBOMs, and GitHub artifact attestations using
     `actions/attest@v4`; pushed images record digest artifacts and receive
-    digest-based provenance/SBOM attestations. The newly introduced Anchore/Syft
-    SBOM action is pinned by commit SHA; broad pinning of existing first-party
+    digest-based provenance/SBOM attestations. Native package SPDX scans now
+    use the exact package file and are verified against the sidecar; full
+    release assembly combines all 11 uniquely named per-row BuildKit statements
+    into one aggregate statement. The Anchore/Syft SBOM action is pinned by
+    commit SHA; broad pinning of existing first-party
     and Docker actions remains a future hardening pass if required by release
     policy.
 
@@ -198,14 +213,12 @@ strategy.
   - Add a release-asset promotion step before public package distribution so the
     exact packages, `SHA256SUMS`, `.sha256` files, SBOMs, image digest records,
     and attestation references are retained for the release support window.
-  - Final result: `images-release.yml` now adds publish-only
-    `ensure-github-release` and `publish-release-assets` jobs for `push=true`
-    runs. The release job creates or reuses the matching GitHub Release in this
-    repository and records the upstream `mesh_source_sha`; the promotion job
-    downloads the row's native package and image digest artifacts, stages unique
-    release asset names for package manifests/SBOMs and image SBOMs, writes
-    row-specific attestation verification notes, and uploads everything with
-    `gh release upload --clobber` using job-scoped `contents: write`.
+  - Final result: `images-release.yml` assembles the complete native release
+    during unfiltered dry runs as well as publish runs. Publication creates a
+    new immutable release without clobbering assets, or treats a pre-existing
+    release as a no-op only after its tag target, title, body, draft state, exact
+    asset names, and GitHub-computed asset digests match the local assembly.
+    Any mismatch fails closed.
 
 - [x] Consider package signing before public distribution.
   - Sign `.deb`, `.apk`, and `.pkg.tar.zst` artifacts if distributing outside GitHub Releases.

docs/package-signing.md

@@ -6,6 +6,13 @@ verification notes, and composition provenance naming the immutable host and
 runtime digests. Do not publish unsigned artifacts through apt, apk, pacman, or
 Homebrew package repositories.
 
+The SBOM is valid release evidence only when its SPDX file subject is the exact
+native package basename and includes the package's verified SHA-256. Directory
+inventory alone is insufficient. The canonical release `provenance.json` must
+contain all 11 native package name/digest subjects and retain the corresponding
+uniquely named per-row BuildKit statements. The GitHub artifact attestation is
+generated once from the same 11-subject checksum list.
+
 ## Repository signing requirements
 
 Before enabling distro package repositories, define and test the signing path for
@@ -34,6 +41,9 @@ each package format:
 - Keep GitHub Release assets, host/runtime/product checksums, SBOMs,
   composition provenance, attestation references, image digests, and signatures
   for the full support window of each `mesh-llm` release.
+- Never replace or clobber a versioned release asset. Treat an exact existing
+  release as a no-op and any metadata or digest difference as a release incident
+  requiring review and a new version.
 
 ## Release gate
 

docs/publishing.md

@@ -2,7 +2,9 @@
 
 The release workflow has four publication states:
 
-1. `dry_run=true`: all required archive, package, image, and Homebrew validation runs; publication is forcibly disabled.
+1. `dry_run=true`: all required archive, package, image, and Homebrew validation
+   runs; publication is forcibly disabled. An unfiltered dry run also assembles
+   and verifies the exact release asset set and aggregate package provenance.
 2. `dry_run=false,publish_images=true`: validated OCI images are pushed to GHCR and receive build-provenance attestations.
 3. `dry_run=false,publish_release_assets=true`: native packages, checksums, SPDX SBOMs, and the rendered Homebrew formula are attached to a `packaging-v<version>` release in this repository.
 4. `dry_run=false,publish_npm=true`: the install-tested `@mesh-llm/sdk`
@@ -11,6 +13,29 @@ The release workflow has four publication states:
 
 Non-npm publish jobs use the `release` GitHub environment, whose deployment policy accepts only `main`, while npm publishing uses the separate `npm` environment described below. Publish jobs have job-local write permissions, and all build and validation jobs are read-only. Add required reviewers when the repository plan supports environment reviewers. The upstream tag must already have a non-draft GitHub Release, the repository must be exactly `Mesh-LLM/mesh-llm`, the ref and version must match, and the tag is resolved to an immutable commit SHA for provenance labels.
 
+Filtered runs are validation-only. If any native variant, platform, or npm lane
+filter is present while a publication switch is enabled, planning fails before
+build or publication. Native release publication therefore always represents
+the checked-in complete 11-row package matrix.
+
+Every enabled native package row preserves its BuildKit statement under an
+artifact-specific filename, scans the exact package file into SPDX, and verifies
+that the SPDX file subject names and hashes that same package. The current
+11-row release matrix contains `.deb` and `.pkg.tar.zst` packages only; the same
+exact-subject rule applies to `.apk` if an Alpine row becomes release-enabled.
+The release assembler rejects missing, duplicate, or mismatched inputs, then
+emits one `provenance.json` in-toto statement with all 11 package
+name/SHA-256 subjects and the 11 per-row BuildKit statements. It also emits one
+aggregate `SHA256SUMS`; neither aggregate hashes itself.
+
+Versioned package releases are immutable. A publish run either creates the
+release once, without `--clobber`, or does nothing when an existing release has
+the exact expected tag target, title, body, non-draft/non-prerelease state,
+asset-name set, and GitHub-computed asset SHA-256 values. Missing or extra
+assets, absent digests, changed bytes, metadata drift, API errors, or tag drift
+fail the job. Do not repair a partial or mismatched release in-place; preserve
+it as evidence and publish a corrected upstream/package version after review.
+
 GitHub Release assets, GHCR, npm, and the
 [`Mesh-LLM/tap`](https://github.com/Mesh-LLM/homebrew-tap) Homebrew tap are the
 enabled public channels. Do not create apt, apk, or pacman repositories until

docs/release-checklist.md

@@ -12,6 +12,9 @@
 ## Dry-run QA
 
 - [ ] A full `dry_run=true` workflow succeeds with publish jobs skipped.
+- [ ] The unfiltered dry run assembles exactly 11 package rows and 47 release
+  assets. `SHA256SUMS` exactly covers every non-manifest asset, and the single
+  aggregate `provenance.json` contains 11 unique package name/SHA-256 subjects.
 - [ ] Every native package passes metadata inspection, exact filename/checksum
   checks, package-manager installation, `mesh-llm --version`, `--help`, and
   `mesh-llm runtime list` without GPU passthrough. The package owns the
@@ -38,6 +41,15 @@
 - [ ] A reviewer confirms the selected publish switches and the `release` environment gate.
 - [ ] GHCR tags match `docs/tagging.md`; pushed digests receive provenance attestations.
 - [ ] Package release assets contain exact packages, SHA256 manifests, SPDX SBOMs, and the rendered formula.
+- [ ] Every package SPDX document names the exact `.deb` or `.pkg.tar.zst`
+  basename and its verified sidecar SHA-256. Every uniquely named per-row
+  BuildKit statement names the same subject before aggregate assembly.
+- [ ] No variant, platform, or npm lane filter is present on a publish run.
+- [ ] A new `packaging-v<version>` release does not already exist. If it does,
+  the workflow may no-op only when tag target, title, body, state, exact asset
+  names, and GitHub asset digests all match; otherwise publication must fail.
+- [ ] Release upload does not use `--clobber`, and the post-create API check
+  proves the published asset set is byte-for-byte identical to the assembly.
 - [ ] If `publish_npm=true`, npm provenance names `mesh-packaging` and the
   version has the expected `latest` or `next` dist-tag.
 - [ ] No native runtime bundles or manifest are republished here.

scripts/image-matrix.ts

@@ -65,6 +65,7 @@ export type Config = {
 
 export type MatrixRow = {
   artifact_id: string;
+  package_file: string;
   upstream_artifact_id: string;
   upstream_asset_name: string;
   upstream_checksum_name: string;
@@ -206,6 +207,14 @@ function nativePackageArtifactName(version: string, variant: Variant, arch: stri
   return `mesh-llm-package-${version}-${artifactId(variant, arch)}`;
 }
 
+export function nativePackageFile(versionInput: string, variant: Variant, arch: string): string {
+  const version = normalizeVersion(versionInput);
+  const distro = requiredString(variant.distro);
+  const backend = requiredString(variant.backend);
+  const packageFormat = requiredString(variant.package_format);
+  return `mesh-llm-${version}-${distro}-${arch}-${backendSuffix(backend, variant.backend_version ?? "")}.${packageFormat}`;
+}
+
 function expectedFlavor(backend: string, backendVersion: string): UpstreamFlavor | "" {
   if (backend === "cpu" || backend === "rocm" || backend === "vulkan") return backend;
   if (backend === "cuda") {
@@ -405,6 +414,7 @@ export function matrixRows(
       const asset = upstreamAssetName(version, targetTriple(platform), flavor);
       rows.push({
         artifact_id: rowArtifactId,
+        package_file: nativePackageFile(version, variant, arch),
         upstream_artifact_id: upstreamArtifactId(asset),
         upstream_asset_name: asset,
         upstream_checksum_name: `${asset}.sha256`,