[nood-co1]

Stage-2 g0 (MLA + indexer) disk restore at 3584 boundaries (GLM53_KV_OFFLOAD_RESTORE, default off) + the stage-3 mamba differential harness

Stacks on #99 (pr/kv-offload-store-only): this branch is
pr/kv-offload-mla-restore = #99 + additive commits; review the diff against
that branch. Codex gates: advisory on the design BEFORE coding
(CODEX-OFFLOAD2-PLAN.md: DO-NOT-PROCEED, 5 BLOCKER + 10 MAJOR — every finding
adopted or rebutted in writing before the build), adversarial review of the
final diff (CODEX-OFFLOAD2-REVIEW.md: DO-NOT-SHIP, 1 BLOCKER + 8 MAJOR +
2 MINOR — every finding adopted or rebutted; fixes in the review-fixes
commit; confirm pass appended there). Host suites green (pytest 40 + six
standalone runners, 0 failures). Live
receipts are posted in the thread; rows 1 and 4 are captured, row 5 is
partial, and rows 2/3 remain open for the reasons in the Receipts section
below.

The honest headline

A group-0-only restore yields ZERO hit uplift on GLM-5.3-Flash itself, by
design — and this PR says so up front. The connector's reconciled external
hit is the min across all KV-cache groups; the four mamba (KDA) groups have
no restore source in this stage. Independently, the core scheduler's
invalid-block path is single-group (pinned fixture
tests/fixtures/image_487ecf187_core_sched_scheduler.py:2954-2955, the
(req_block_ids,) = …get_block_ids(req_id) under the hybrid-allocator TODO)
— a load-failure report on the 7-group layout would crash the core. The
restore path therefore requires exactly one TOTAL KV-cache group and is
INERT on this deployment with one boot log line naming the reason;
serving with GLM53_KV_OFFLOAD_RESTORE=0 (default) matches #99 at the
serve-argv level (live capture); byte-for-byte runtime parity is a host-test
property, not a live receipt.

What stage 2 delivers instead: the restore machinery and host-side tests.
The 2026-09-02 dense single-full-attention-group probe live-receipted the
path through scheduler lookup and disk-job creation, but not successful
worker restore. It is not the exact MLA+indexer shape of g0, and the T4
recompute contract remains unproven live. These are the prerequisites
stage 3 (mamba restore, the actual TTFT win) needs underneath it.

What this ships

overlay/patch_kv_offload_restore_g0.py (five patch targets)

• Lookup from the on-disk manifests, keys/hashes only: each rank's store
  writer announces manifest availability over the worker→scheduler metadata
  channel (+ validated publish / - retention supersede / F write
  failure); the scheduler ANDs + across ranks (plan §4 C2 — "workers
  report it; the scheduler takes the min across ranks") and never touches
  the filesystem. Contiguity is content-addressing: block hashes are
  prefix-chained, so boundary-hash equality implies the whole chain.
  The per-job store failure bit is on the wire and load-bearing: a
  boundary whose g0 chunk failed to write on any rank is never offered
  (closes the recorded stage-1 limit).
• Delay-cache allocation + disk load jobs: a disk hit skips
  manager.prepare_load entirely (CPU staging is a bounce, not a tier);
  the load job's src_spec is a versioned plain dict with entries aligned
  1:1 to the dst blocks; WAITING_FOR_REMOTE_KVS and the cache_blocks
  re-entry are the stock core chain.
• Worker pread with full verification: synchronous per-chunk
  pread → header identity (namespace/hash/group/spec/rank/world) → payload
  CRC → exact segment-table equality against this rank's own spec
  derivation → scatter into the GPU canonical tensors. Restore
  concurrency = 1. Manifest re-validated on each rank at load time.
• T4 (stage-2 contract): any per-(chunk, rank) failure zero-fills the
  failed chunk and every later chunk's dst blocks, reports their ids via
  get_block_ids_with_load_errors (facade override; the model-runner mixin
  consumes it into invalid_block_ids), and the job still completes — the
  core's kv_load_failure_policy=recompute truncates at the failed chunk
  and recomputes. All roads lead to reprefill, never a served half-restore.
• T2: request.glm53_restored_boundary (end-exclusive) rides into
  reachable_boundaries in the single_type manager's cache_blocks, so
  sparse retention masks can never silently drop a restored boundary's
  re-entry (dense masks unaffected; the anchor also applies verbatim to
  upstream's tree — retention exists at HEAD).
• Preemption/reset hygiene: neither the preemption flush nor
  _build_store_jobs can hit the stock is_store asserts on a disk load
  job (flush skips it; store creation is deferred while a load is
  registered); the single-flight gate self-heals; stale completions are
  discarded by the existing _stale_job_threshold; per-rank writer
  generations are fenced; every registry/queue is bounded fail-closed.
• S1: boundary manifests now also publish on layouts with no recurrent
  groups (hybrid layouts byte-identical to Stage-1 store-only disk KV tier: eligible-group scoping + worker-local per-rank store (GLM53_KV_OFFLOAD, default off) #99) — without this the probe
  layout had nothing to look up.

Launcher

GLM53_KV_OFFLOAD_RESTORE=1 is accepted IFF GLM53_KV_OFFLOAD=1 (refused
fail-closed pre-stop otherwise, zero docker/ssh calls); the restore overlay
rides the same artifact-guard / --check-injected / scp / mount / order
machinery as the stage-1 pair (order pinned scope → store_local →
restore_g0); offload_prompt_only:false and every stage-1 knob unchanged.

Stage-3 mamba cross-boot differential harness (design + fixtures ONLY)

tests/_mamba_differential.py + tests/test_mamba_differential_harness.py:
park-at-boundary fixtures with stubbed KDA state tensors at the receipted ABI
(conv (10,12288) bf16 + temporal (32,128,128) fp32, real page 2,342,912 B,
9/9/8/8 layers, num_spec=7 in every header, padding never on disk) and the
exact bit-comparator stage 3 will run (a flipped bit localizes to its
(group, layer, tensor)). tests/stage3_mamba_differential_protocol.sh pins
the live protocol (step 0 = receipt the still-open mamba state-index
convention) and refuses to run on this branch. This is an ABI/codec fixture
comparator, not stage-3 evidence — no live state was captured.

New pinned fixtures (read-only captures, sha256 in tests/fixtures/README.md)

offloading_connector.py and single_type_kv_cache_manager.py (patch
targets) plus the core sched/scheduler.py (receipt only, not patched) —
captured from the live container running the identical vllm tree
(0.1.dev20051+g487ecf187; the four actual offloading patch-target fixture
sha256s were re-verified against it byte-exact; the core scheduler capture
is a non-target receipt — see the 2026-09-02 receipts comment's P3
exception).

Tests (host-side, all green: pytest 40 + 6 standalone runners)

file	covers
tests/test_kv_offload_restore_g0.py	patcher hygiene on all five targets (idempotence/tamper/store-less refusal/--check-injected); patched-runtime scheduler: eligibility matrix (7-group INERT + named reason, single-group ACTIVE, one-eligible-plus-excluded INERT), AND-across-ranks registry (+/−/F), deepest-boundary lookup, alignment guard, single-flight gate (defer/self-heal/completion/reset), disk-job shape with ZERO manager calls and preserved stage-1 bookkeeping, preemption load-job guard, zero-cow manifest candidates, RESTORE=0 parity; loader: byte-exact restore round-trip against real writer output, failure matrix (CRC/missing file/namespace/manifest-gone/wrong-rank/null-block) with zero-fill + invalid ids + job-always-completes, earlier-chunks-kept/suffix-abandoned; event channel: emission, validated dedup, event-only meta flow, aggregate + legacy meta, pickle; facade drain; T2 on the patched single_type text incl. the mamba mask end-exclusive convention (3583/3584/7168)
tests/test_mamba_differential_harness.py	ABI field checks (shapes/dtypes/sizes/num_spec/padding-never-on-disk, drift flagged), comparator (clean/restored-copy/single-bit localization/full divergence)
tests/test_kv_offload_restore_venv.py	feature-detected coordinator checks against the CPU vLLM venv: T2 anchor applies to the real tree; connector load-errors API present
tests/test_launcher_kv_offload.py (extended)	RESTORE=1 without OFFLOAD=1 refused pre-stop (zero host calls); OFFLOAD=1+RESTORE=1 accepted; truncated restore overlay blocks pre-stop; overlay order scope→store→restore; RESTORE replayed identically to both ranks

Receipts (status 2026-09-02; history in the thread's receipts comments)

Rows 1 and 4 are captured. Rows 2/3 were run live on a purpose-built
single-group probe (2026-09-02 receipts comment) and remain open on three
branch gaps that run found: (A) the v1 loader's fail-closed mapped-layout
guard means no dense model can exercise the restore chain (ACTIVE proves
scheduler eligibility, not worker restorability); (B) the launcher leaves
kv_load_failure_policy at the image default fail, so load failures 500
the request instead of the designed degrade-to-recompute; (C) no
per-boundary failure suppression exists after a failed load. Each needs a
reviewed branch commit before rows 2/3 can close:

1. Inert-on-live receipt: boot the 7-group deployment with
   GLM53_KV_OFFLOAD=1 GLM53_KV_OFFLOAD_RESTORE=1; the boot log shows
   [glm53-kv-offload-restore] restore INERT on this layout: layout has 7 KV-cache groups (5 eligible)… on the head; serving probes green;
   decode/prefill ranges overlap Stage-1 store-only disk KV tier: eligible-group scoping + worker-local per-rank store (GLM53_KV_OFFLOAD, default off) #99's knob=1 sample, but this capture does
   not establish the body's "within noise" criterion (no decode-rate
   comparison was captured).
2. Probe fork-trace receipt (single-full-attention-group probe config,
   plan §5/F5) — OPEN, blocked on gap A (then B): park a session past
   the store cascade; new request hits the disk lookup → async-load path
   evidenced by WAITING_FOR_REMOTE_KVS state occupancy (the core logs the
   in-state line at DEBUG only and never logs the transition pair), the
   disk load job N restored … line on BOTH ranks, zero load errors, and a
   follow-up request served as an ordinary GPU prefix hit with zero disk
   traffic. 2026-09-02: everything upstream of the worker pread is
   live-receipted on the dense probe (cascade, lookup/offer, state entry,
   job creation on both ranks); the pread stage requires an MLA-layout
   model or v1.1 inverse-scatter support.
3. T4 fault receipts — OPEN, blocked on gaps B/C (the scheduler
   recompute path applies only when kv_load_failure_policy="recompute" is
   explicitly configured; the current launcher does not set it and has not
   proven live reprefill): corrupt one
   chunk file (bit flip) and delete one manifest on one rank → restore
   degrades to recompute (invalid-block log + recompute), output equals the
   cold-path control, fresh-session control green. The null-block exclusion
   is host-suite evidence (test_kv_offload_restore_g0.py); live logs carry
   only counts. 2026-09-02: the worker half (fail-closed, invalid-block
   report, zero-fill) is live-receipted; the scheduler recompute half needs
   kv_load_failure_policy="recompute" pinned by the launcher plus
   per-boundary failure suppression.
4. Kill-switch receipts: RESTORE=1 without OFFLOAD=1 refused
   pre-stop with rc=2 and the serving container's StartedAt unchanged
   (live); the zero docker/ssh-call property is established by launcher stub
   tests, not the live transcript. RESTORE=0 adds no restore arguments;
   the live capture establishes serve-argv parity only, not byte-for-byte
   runtime parity with Stage-1 store-only disk KV tier: eligible-group scoping + worker-local per-rank store (GLM53_KV_OFFLOAD, default off) #99.
5. Event-channel receipt — PARTIAL (strengthened 2026-09-02): the
   overlay logs neither publish counts nor the scheduler AND, so the
   literal wording is not observable. Live-receipted instead: writers up on
   both ranks and a real boundary offered from disk with the load job
   created on BOTH ranks — per the overlay source a boundary is offered
   only after the scheduler ANDs + events from both ranks, so this is
   source-backed indirect evidence the cross-rank registry ANDed live.

---

Submitted by nood-co1 on behalf of Blockbrain Labs (https://x.com/blockbrain_labs, GitHub org blockbrain-ai). Measured on our 2× NVIDIA DGX Spark deployment.

[nood-co1]

Live receipts — stage-2 restore, inert-on-live (server window 2026-09-01/02)

Same window and deploy set as the #99 receipts (staged deploy/dogfood-next4; the RESTORE boots ran @ 032ee21, which includes this branch @ 22e1817 — i.e. stacked on #99 plus its window fix 087b5ea). Condensed/redacted excerpts; raw transcripts kept in our receipts mirror. Two receipt rows are pending and one partial, with reasons stated.

1. Inert-on-live receipt — CAPTURED

Boot with GLM53_KV_OFFLOAD=1 GLM53_KV_OFFLOAD_RESTORE=1 on the live 7-group hybrid:

(EngineCore pid=2288) INFO 09-01 19:18:36 [scheduler.py:743] [glm53-kv-offload-restore] restore INERT on this layout: layout has 7 KV-cache groups (5 eligible): g0-only restore yields no external hits on a hybrid layout by design
health=200

This proves the group-0 disk-restore lookup is disabled by the seven-group eligibility gate on this layout; it does not exercise disk loading, the worker restore path, T4 handling, or event transport (those are rows 2/3/5).

Serving rates vs #99's knob=1 boot (same cold ~26K-prefill probe, 6 reps each arm, reps 4–6 quoted for both arms):

• knob=1, RESTORE=0: 26.53–27.90 s
• knob=1, RESTORE=1: 26.51–28.37 s

The reps-4–6 ranges overlap but are not identical; this cold-prefill sample does not independently establish the body's "within noise" criterion (no decode-rate comparison was captured).

4. Kill-switch receipts — CAPTURED

• GLM53_KV_OFFLOAD_RESTORE=1 without GLM53_KV_OFFLOAD=1: refused pre-stop, rc=2, with the serving head container's StartedAt unchanged (live transcript: NOTHING-STOPPED). The zero-docker/ssh-call property is proven by the stubbed launcher tests, not by this live transcript.
  GLM53_KV_OFFLOAD_RESTORE=1 requires GLM53_KV_OFFLOAD=1 (the restore path reads the store tier)
• RESTORE adds no serve argv either way: the NUL-split vllm serve argv of the RESTORE=1 boot is identical to the RESTORE=0 knob=1 boot (diff of the two 60-line captures → ARGV-IDENTICAL-B-vs-C). The supplied diff proves serve-argv parity only, not byte-for-byte runtime behaviour.

2, 3. Probe fork-trace and T4 fault receipts — PENDING, with reason

The restore path requires exactly one TOTAL KV-cache group (the PR's own eligibility rule). This branch ships no runnable single-full-attention-group probe config: exercising it needs a different model served under a purpose-built config (single group, no drafter, no mamba), which cannot be derived from this repo's launcher without designing a new deployment — out of scope for a window that must return the production pair. Rows 2 (fork-trace: WAITING_FOR_REMOTE_KVS → disk load job → GPU prefix hit) and 3 (T4: corrupt chunk / deleted manifest → zero-fill + invalid_block_ids + recompute, fresh-session control) therefore remain open — and per the body they gate merge-eligibility until captured. The wire path is covered host-side by tests/test_kv_offload_restore_g0.py (byte-exact restore round-trip against real writer output; failure matrix with zero-fill + always-ack), which is evidence of code behaviour, not of live serving.

5. Event-channel receipt — PARTIAL, with reason

From the source: the worker appends manifest events on publication and forwards them in worker metadata even when restore is INERT, and the scheduler consumes them into the registry; only lookup() short-circuits on the eligibility gate. But the overlay logs neither event counts nor registry state, and with lookup inert nothing ever surfaces the registry — so the supplied boot logs cannot evidence event flow or cross-rank AND-ing (grep -ci manifest = 0 on both ranks, pasted in the transcript). Separately, on-disk boundary manifests exist and verify on both ranks (GC full-CRC across a restart: manifests=16 chains=8 corrupt=0 per rank) — that is store/GC evidence, not event-flow evidence. The flowing-events receipt needs the same single-group probe as rows 2–3, where a disk hit makes the registry observable, and stays open with them.

---

End state: stores deleted on both ranks, production deploy (next3 972370a, knobs absent) restored and healthy. The stage-1 receipts, including the launcher defect found and fixed live (PYTORCH_CUDA_ALLOC_CONF × OffloadingConnector, 087b5ea), are in the #99 thread.

[nood-co1]

Single-group probe receipts — rows 2/3 stay open, and we found out why (test-rig window 2026-09-02)

We built and ran the single-full-attention-group probe this PR's pending rows called for. The probe booted ACTIVE and ran end-to-end up to the worker restore stage — where it hit a fail-closed guard in this branch's own v1 loader: restore is blocked at the worker pread stage for this dense mapped layout. Rows 2 and 3 therefore remain open, now with a precise, receipted reason and three branch gaps filed below. Row 5's evidence is substantially strengthened. Raw transcripts in our receipts mirror; condensed excerpts here.

Probe config (disclosed dense deviation). pr/kv-offload-mla-restore @ c5dd69d (this PR's exact bytes), same fat-e2 image as production, TP=2 across a 2×DGX-Spark pair (test rig), Qwen/Qwen3-0.6B @ c1899de, SPEC_METHOD=none, --block-size 64 --num-gpu-blocks-override 192 --seed 0 ⇒ exactly one TOTAL full-attention group. The probe is dense GQA, not g0's MLA+indexer shape: no suitable vLLM-loadable small MLA model was available in the checked assets (the kit found none under ~30 GB), so dense was the planned honest minimum — on the premise that the loader does not branch on spec kind. That premise turned out to be false at exactly one code site (finding A below), which is itself the probe's main result.

Boot gate + eligibility — selected live excerpts (module paths abbreviated by vLLM itself; both ranks shown):

(EngineCore pid=1585) INFO 09-02 00:19:51 [...offloading/scheduler.py:747] [glm53-kv-offload-restore] g0 disk restore ACTIVE (single full-attention group 0, chunk 64 tokens, 2 workers)
(Worker_TP0 pid=1823) INFO 09-02 00:19:09 [...offloading/config.py:144] [glm53-kv-offload-scope] eligible groups: [0] of 1 (drafter_included=False)
(Worker_TP0 ...) [glm53-kv-offload-store] writer up: rank 0/2 base=/data/glm53-kv-offload/glm53kv_..._r0 eligible=[0]
(Worker_TP1 ...) [glm53-kv-offload-store] writer up: rank 1/2 base=/data/glm53-kv-offload/glm53kv_..._r1 eligible=[0]

What ran and was proven live:

1. Store cascade on the single-group layout: 4160-token park (65 full 64-token chunks) → deepest boundary manifest 4160 on BOTH ranks' NVMe; store metrics on-line (kv_offload_store_bytes=469762048 … store_size_count=4 and growing).
2. Eviction fallback → real disk lookup → cross-rank offer: the reset endpoint returned 404 on this build, so the kit's flood fallback ran; the re-request then reached the disk-load path — the scheduler's disk lookup offered a 2-chunk (128-token) boundary and created disk load job 7 on BOTH ranks. (No direct LRU-eviction counter is logged; the 2-chunk job size is the observable.) Per the overlay source, a boundary is offered ONLY after the scheduler ANDs + events from both ranks — a job actually created from that lookup is source-backed indirect evidence the cross-rank registry ANDed live (there is still no counter log; see row-5 wording note).
3. WAITING_FOR_REMOTE_KVS occupancy, verbatim (DEBUG boot): cmpl-9b34e0e20bcf708c-0-a4e31ccf is still in WAITING_FOR_REMOTE_KVS state. (The core never logs the transition pair the body row implies — the DEBUG line only fires when a still-loading request is skipped by a later pass. Body row worded accordingly, see amendment.)
4. The v1 fail-closed layout guard fired, symmetrically on both ranks (exact lines, both worker ranks shown; the ... in the module path is vLLM's own abbreviation):

(Worker_TP0 pid=1823) WARNING 09-02 00:22:55 [distributed/.../offloading/worker.py:1410] [glm53-kv-offload-restore] disk load job 7 FAILED at chunk 0/2 (ValueError: canonical-mapped layout not restorable (v1)): reported 2 invalid block(s), zero-filling the suffix (recompute)
(Worker_TP1 pid=1598) WARNING 09-02 00:22:55 [distributed/.../offloading/worker.py:1410] [glm53-kv-offload-restore] disk load job 7 FAILED at chunk 0/2 (ValueError: canonical-mapped layout not restorable (v1)): reported 2 invalid block(s), zero-filling the suffix (recompute)
(EngineCore pid=1585) ERROR 09-02 00:22:55 [v1/core/sched/scheduler.py:2983] Failing 1 request(s) due to KV load failure (failure_policy=fail, 128 tokens affected). Request IDs: {'cmpl-9b34e0e20bcf708c-0-a4e31ccf'}

The worker's generic load-failure handling was live-receipted for this mapped-layout guard (fail-closed, invalid-block report, zero-fill); this did not exercise row 3's bit-flip/deleted-manifest cases. The request then died with an API 500 instead of recomputing — finding B.

Branch gaps found (all three verified in source; each needs a reviewed branch commit before rows 2/3 can close)

A. The v1 restore loader cannot restore any canonical-mapped (dense fused-KV) layout — by its own fail-closed design. overlay/patch_kv_offload_restore_g0.py (restore path, mapped-ref guard): any GPU group data ref with a non-None canonical mapping raises canonical-mapped layout not restorable (v1), because a raw row copy is not the inverse of the store-side gather. Dense GQA under FLASH_ATTN is canonical-mapped; g0's MLA latent layout is what v1 supports. Consequences: (i) the boot's ACTIVE line proves scheduler eligibility, not worker restorability — the design doc should say so; (ii) the host suite's happy path uses mapping=None fixtures, so this was invisible to the 40-test suite; (iii) rows 2/3 cannot be captured on this dense mapped layout — they need either a small MLA-layout model (none was available in the checked assets) or a reviewed inverse-scatter (v1.1) implementation.

B. The launcher never sets kv_load_failure_policy; the image default is fail. So live, ANY load failure kills the request with a 500 — including the T4 faults row 3 is supposed to demonstrate as degrade-to-recompute. The core supports recompute (scheduler.py gates on it and has the Recovered from KV load failure … rescheduled path). RESTORE=1 boots must pin kv_load_failure_policy="recompute" in the launcher's --kv-transfer-config.

C. Recompute alone may still not converge: no per-boundary failure suppression. After a load failure the overlay clears its single-flight gate but does not quarantine the failed boundary in the cross-rank registry — the scheduler can be offered the same permanently-failing suffix again. The branch needs reviewed per-request/boundary failure suppression plus an integrated scheduler test before row 3's "output equals cold control" can be captured even on a restorable layout.

Row status after this window

• Row 2 (fork-trace): OPEN — blocked on A (and then B). Everything upstream of the worker pread is now live-receipted.
• Row 3 (T4 faults): OPEN — blocked on B/C (the worker's generic load-failure handling is live-receipted — fail-closed + invalid-block + zero-fill — but the bit-flip/deleted-manifest cells were not exercised; null-block exclusion remains host-suite evidence).
• Row 4 (kill-switch): re-receipted on this rig — GLM53_KV_OFFLOAD_RESTORE=1 alone refused pre-stop, rc=2, serving container's StartedAt unchanged.
• Row 5 (event channel): PARTIAL, substantially strengthened — writers up on both ranks + a real boundary offered from disk (job created on both ranks). Still no publish-count/AND log exists, so as literally worded it stays partial; the body row should be amended to the observable evidence.

Also disclosed: (i) boots carried a local env passthrough in start.sh (VLLM_LOGGING_LEVEL, PYTHONHASHSEED); overlay/connector bytes unmodified. (ii) The pinned core-scheduler receipt fixture does not sha-match the pristine image — the fixture was captured from a container where an unrelated runtime overlay (decode-floor v2) had already patched scheduler.py; the diff is purely additive, the invalid-block receipt lines are byte-identical at a +119-line shift, and all four actual patch-target fixtures match the image byte-exact. The fixtures README should pin pristine-image bytes for non-target receipts. (iii) Cross-boot caveat, relevant to stage 3 and any future cross-boot restore claim: this image generation seeds NONE_HASH from os.urandom(32) whenever PYTHONHASHSEED is unset (kv_cache_utils.py @ 487ecf187), so on-disk chunk/manifest keys drift across boots even under sha256 and the store namespace fences the algo but not the seed. A launcher pin + namespace fence is owed on this branch. All receipts above are same-boot and unaffected.

Boot provenance. Three boot attempts failed pre-receipt (a stale baked self-check unrelated to the branch; a root-owned weight-cache permission error from the download fallback; and vLLM's zero-tolerance memory-profiling assertion racing the prior tenant's teardown on unified-memory hardware). Attempt 4 was the single advisory-authorized retry: prior logs retained, containers/stores reset first, no rsync/download/other tenant active, and three consecutive 10 s MemAvailable samples within 200 MiB on both nodes before booting — a mitigation for the profiling race, not proof against it. Exact arguments: GLM53_KV_OFFLOAD=1 GLM53_KV_OFFLOAD_RESTORE=1 VLLM_LOGGING_LEVEL=DEBUG ./start.sh restart.

Every deviation and boot retry in this window ran under adversarial advisory gates (three rig advisories + this comment's posting gate); dispositions archived with the receipts.

---

Submitted by nood-co1 on behalf of Blockbrain Labs (https://x.com/blockbrain_labs, GitHub org blockbrain-ai). Measured on our 2× NVIDIA DGX Spark deployment.

[nood-co1]

Updated to current main (e25aa70) the stacked way: the refreshed #99 branch (26ca3cd, which now contains main) was merged INTO this branch — merge commit ca2e31e, plain push, no force-push; the full tested stack (stage-1 SHAs, 22e1817, c5dd69d, the stage-2/3 review chain) remains ancestors. The merge was conflict-free: main's changes arrived pre-resolved through #99, and this branch's stage-2 delta (restore knob unlock, patch_kv_offload_restore_g0.py, stage-3 harness) did not overlap them. Verified post-merge: upstream/main and #9926ca3cd are both ancestors; GLM53_OVERLAY_ORDER reads …scope → store_local → restore_g0 → xgrammar → kpool → spinwait → indexer_workspace → ablit (main's new overlays in main's own position; the documented build-time exclusion of patch_exl3_fat_kernel.py carries over — see the #99 update comment for the reasoning); the restore-requires-store guard is intact. Re-run green: bash -n, the branch's kv-offload + restore + stage-3 harness host tests, full host suite 22/26 (4 known env fails identical on upstream/main). Also folded into the update (from its review): three stale stage-1 comment/docs lines still said GLM53_KV_OFFLOAD_RESTORE=1 is refused — corrected to the stage-2 contract (RESTORE=1 valid, requires GLM53_KV_OFFLOAD=1); the guard itself was already stage-2 and is unchanged. Host suites re-run green post-merge. No new live receipts were taken for this update: the pre-merge receipts and the T4 probe findings (including the v1-restore-loader amendment and the open degrade-to-recompute / failed-boundary-suppression items) stand exactly as posted — this update changes none of that, it only brings the branch onto current main.

[plotarmordev]

Community triage note (full sweep: #147): one deployment-relevant finding to flag — on this model's 7-group hybrid KV layout the restore path here is inert: the mamba groups have no restore source and the core scheduler load-failure path is single-group, which matches your own ZERO-hit-uplift headline. As of today #58 is the only offload PR with a working measured restore. Tagged triage/skip for this deployment.

[plotarmordev]

Thank you for the most honest PR in this queue — the adversarial-review process, the live receipts, and especially the single-group probe that chased down why rows 2/3 stayed open are a model for how this work should be reported. We're closing for this deployment on the strength of your own headline: on this model's 7-group hybrid KV layout the restore path is inert — the mamba groups have no restore source and the core scheduler load-failure path is single-group — so there is zero hit uplift to adopt, and the staged rewrite's allocator surface isn't free. This is a layout-mismatch close, not a quality judgment.

The fair way back: reopen (or a fresh stage PR) when a stage actually restores the mamba/hybrid groups on this 7-group layout with a hit-uplift receipt on a 2-node TP=2 boot. Until then #58's measured restore covers the offload need, and the cache-capacity work is proceeding via the per-group retention line (#130/#83).