Repository navigation
Conversation
…#54743) The OffloadingConnector cannot boot on this hybrid layout: the image's build_offloading_config builds an offload group config for every KV-cache group and asserts tokens_per_block % tokens_per_hash == 0 across all of them, so the KpoolTailSpec scratch group (block 4, hash grid 64) crashes it at boot. Port our upstream PR vllm-project/vllm#54743 onto the image tree (vllm 0.1.dev20051+g487ecf187, mid-generation: the scheduler NamedTuple already carries group_idx, the config boundary does not): - OffloadingGroupConfig gains group_idx (original boot index, required); - build_offloading_config scopes the groups tuple to eligible groups: prefix-cacheable (via the image's participates_in_prefix_caching name) minus the fork policy exclusion of draft-model attention groups (is_eagle_group, or exact SlidingWindowSpec under an EAGLE speculative config) unless GLM53_KV_OFFLOAD_DRAFTER=1; the divisibility assert still guards eligible groups; one boot line names the eligible set ({0, 2, 3, 4, 5} on this deployment); - scheduler port: group_states sized by the FULL group count, every config<->state pairing through group_idx, _group_config_by_idx for the two subscripts, full-length GPULoadStoreSpec group_sizes/block_indices with zeros for ineligible groups (the worker layout is built from ALL groups and stays unchanged), supports_partial_tail additionally requires every group eligible; - [I]-specific adaptation: a fail-closed single-type unwrap for the image's UniformTypeKVCacheSpecs group wrappers, without which the connector's FullAttentionSpec assert crashes on group 0 and mamba groups lose their alignment/CoW classification. Fail-closed patcher per repo convention: all anchors in all three files preflight before any write; MARK sentinels; idempotent (marks-only already-present check, because patch_kv_offload_store_local.py later edits inside these regions); atomic replace; drift => nonzero. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…53_KV_OFFLOAD) Stage 1 of the disk KV-offload plan: the tier writes, nothing reads it back (restore short-circuited behind GLM53_KV_OFFLOAD_RESTORE=0). The stock fs secondary tier is byte-invalid on this 2-node TP pair: TieringOffloadingSpec builds every secondary tier scheduler-side against the scheduler node's staging memoryview, and rank 1's staging mmap lives on the other Spark - the tier would persist rank-0 bytes plus uninitialised garbage as rank-1 halves. So the disk write moves to the workers: - TransferJob gains an optional glm53_store_meta side-channel (versioned picklable dict): ordered keys aligned with the job's block order plus boundary-manifest candidates (cumulative chunk-hash chains); - each worker, after its GPU->CPU DMA completes, writes its own bytes from its own staging mmap to its own NVMe (per-rank namespaced root; rank = initialized TP rank cross-checked against the connector config, mismatch disables the writer) and only then acks the job - staging stays pinned until every worker acks, so bytes cannot be evicted under the writer; - every chunk gets a versioned header: magic, namespace hash (model, vllm version, build id, TP, KV dtype, hash algo, num_speculative_tokens, per -group real page sizes, KDA state-ABI descriptor), ORIGINAL group index, per-layer segment table (conv/temporal split with shape/dtype/stride for KDA layers), boundary token index, payload CRC; payload = real (unpadded) bytes only - the slot-share padding never reaches disk; - boundary manifests publish per-rank only after all four mamba payloads and the cumulative full-attention chunks are durable (fsync file+dir, tmp+rename); a crash between group writes leaves orphans, never a live half-boundary; a key that failed to write this boot can never be manifested (failed-key ledger); - inline keep-K manifest retention (GLM53_KV_OFFLOAD_KEEP_BOUNDARIES, default 2): a manifest is superseded once it is beyond the K most recent boundaries of EVERY chain containing it; full-attention chunks stay dense; - failure semantics: failed write = unlink + log + lost store; ENOSPC pauses the writer; serving is never blocked on the tier. kv_offload_store_gc.py: standalone verifier/GC (header+CRC walk, orphan and corrupt reporting, manifest cross-reference, keep-K sweep; DRY-RUN default). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…on both ranks - Knobs with caller-export setness capture: GLM53_KV_OFFLOAD (0|1 strict, default 0), GLM53_KV_OFFLOAD_DIR (absolute, no quotes/whitespace; local NVMe on each Spark, mounted rw at /data/glm53-kv-offload on both ranks), GLM53_KV_OFFLOAD_CPU_GB (1..64), GLM53_KV_OFFLOAD_RESTORE (stage-1 hard rule: 1 is REFUSED), GLM53_KV_OFFLOAD_DRAFTER, and GLM53_KV_OFFLOAD_KEEP_BOUNDARIES (0..1024) - all validated inside the pre-stop gate, so a bad knob or a missing/truncated/mis-pointed overlay artifact exits 2 before restart stops the healthy pair. - knob=0 adds NOTHING beyond the replayed env: no --kv-transfer-config, no store mount, no mkdir - serving is byte-identical to a build without the tier. knob=1 appends identical connector JSON to both rank inner scripts (TieringOffloadingSpec, cpu_bytes_to_use, blocks_per_chunk=1, NO secondary tiers - the worker-local writer is the disk tier; the scheduler-side fs tier is byte-invalid on 2-node TP), mounts the store dir on both nodes and ships both overlays over the scp -> /tmp -> mount chain. - GLM53_OVERLAY_ORDER pins one apply list emitted verbatim into BOTH rank inner scripts (the kv-offload pair is order-dependent: scope must precede store_local, which refuses an unscoped tree); the prefix-cache overlay slots from PRs MiaAI-Lab#83/MiaAI-Lab#84 ride along as [ -f ]-guarded entries. - Dockerfile: bake both overlays after the real-tree preflight test; ship the GC tool and tests into /opt/glm53. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d tier
tests/fixtures/: byte-exact pinned captures of the five in-image sources the
overlays patch (vllm 0.1.dev20051+g487ecf187, sha256s in the fixtures
README) - the patchers preflight/apply against the REAL text and the tests
exec the PATCHED modules under a stubbed vllm namespace
(tests/_kv_offload_stub_env.py), so anchor drift or a broken port fails on
the host, not on the server.
- test_kv_offload_scope.py: patcher preflight/idempotence/tamper/half-patch
on the fixtures; eligibility predicate matrix (kpool never, drafter only
under EAGLE context, strict 0/1 env); patched-runtime drive: eligible
{0,2,3,4,5}, original-index keys, full-length group_sizes with zeros at
1/6, num_kv_cache_groups=7, mamba alignment 3584, store-only lookup
short-circuit (manager.lookup never called), no-scratch uniform model
unchanged, misaligned cacheable group still asserts.
- test_kv_offload_store_local.py: header codec round-trip + truncation at
every region + CRC flips; store-job meta 1:1 alignment with block order +
cumulative manifest candidates + TransferJob pickle; writer on a
miniature 7-group layout: real-bytes payloads, KDA conv/temporal segment
split, manifest publish ordering, failed-key ledger, keep-K retention,
ENOSPC pause with no partial files, TP-rank disagreement disable,
delayed-ack drain; GC dry-run/sweep/keep-K incl. cumulative full-attn
protection.
- test_launcher_kv_offload.py: knob guard matrix, pre-stop gate (bad knob /
missing / truncated / mis-pointed overlay => rc=2 with zero docker/ssh
calls), knob=0 no-mount parity, knob=1 both-rank env/mount/scp/JSON-shape
parity with a synthetic one-rank mismatch, overlay order pin.
- test_kpool_tail_slotmap.py / test_xgrammar_termination.py: launcher-wiring
assertions moved to the GLM53_OVERLAY_ORDER list (same shape as the
kv-capacity-log branch); test_warm_restart_stdout.py: the -S JSON-builder
count is now 4 (2 speculative + 2 kv-offload).
- README overlay/test table rows; docs/DESIGN-kv-offload-store-only.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rdening (Codex adversarial pass) Adversarial diff review (CODEX-OFFLOAD1-REVIEW.md) verdict DO-NOT-SHIP; every finding adopted or rebutted in writing: 1 (BLOCKER, adopted) reset_cache() frees ALL staging rows regardless of pending disk work and the very next step can DMA new stores into them, so async mmap reads could tear a payload under a valid CRC. The writer now SNAPSHOTS the job's bytes synchronously at DMA completion (get_finished, before this rank's ack; or right after worker.wait() on the flush/reset path via a new capture hook) and writes from private, budget-capped buffers; acks are never deferred; shutdown/mmap-cleanup coupling is gone. Regression test: staging rows zeroed after capture, disk bytes unchanged; flush-path capture test. 2 (MAJOR, adopted) the launcher JSON now sets offload_prompt_only=false (upstream defaults true and would silently skip decoded tokens); the test stub mirrors the upstream default instead of masking it. 3 (MAJOR, adopted) namespace record is fsynced and VALIDATED against an existing file (mismatch disables the writer); GLM53_KV_OFFLOAD_BUILD_ID is derived from the recipe checkout SHA and replayed to both ranks; dedup and manifest publication header-VERIFY identity (namespace+hash+group) for mamba AND cumulative full-attn chunks, with per-chunk [hash, len, crc] recorded for full groups (plan §4 wording honored). 4 (MAJOR, adopted) temp names gain the writer thread id; the GC scans and sweeps stale *.tmp.*, infers mamba groups from chunk HEADERS (orphans are found even when a crash predates the first manifest), and applies the writer's leaf-walk keep-set so forked chains keep shared divergence ancestors; forked-chain, zero-manifest-orphan and stale-temp tests added. 5 (MAJOR, partially adopted) both patchers gain --check-injected (compiles every injected source standalone) and the pre-stop artifact gate runs it; the residual exposure (an anchor mismatch surfacing only in-container) is the repo-wide overlay contract and is documented, not new to this pair. 6 (MINOR, adopted) the TP-rank fallback now logs its reason and records rank provenance in the namespace record; the cross-check still disables the writer on disagreement, and the two-node R0 receipt covers the fallback path live. Full suite 33 passed; store tests now 60+ checks incl. the snapshot regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…temp GC Codex confirm pass on 32f993e (CODEX-OFFLOAD1-REVIEW.md appendix): findings 1/2/5 PASS, rebuttals accepted, two residuals — both adopted: - RESIDUAL MAJOR: the namespace record appended keep_boundaries/rank_source AFTER hashing but compared the WHOLE record byte-exact on the next boot, so changing retention K (or the rank-derivation path flipping between TP group and config fallback) spuriously disabled the writer. The record now separates {fenced, info}: the fenced portion alone is hashed and exact-compared; info fields are diagnostics. An unparseable record still fails closed. Tests C22b-d (K change reuses the store; corrupted record disables). - RESIDUAL MINOR: namespace-record temps live BESIDE the per-rank base (the store root); the GC now scans that root non-recursively for glm53kv_*.json.tmp.* in addition to the recursive base scan. Test D9/D10 updated to plant and sweep one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vLLM's VllmConfig validation refuses OffloadingConnector together with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (the VMM allocator can remap KV-cache virtual addresses onto different physical pages under pinned/registered KV memory). The launcher exported the value unconditionally to both ranks, so the first GLM53_KV_OFFLOAD=1 boot failed fast at config validation — found live in the MiaAI-Lab#99 receipt window; the check is unreachable from the host-side stubs. knob=1 boots now drop the env on both ranks (torch default allocator, logged); knob=0 boots keep the stock entry — container env byte-identical to a build without the tier. Launcher test: C3b (knob=0 keeps it, both ranks) + D1b (knob=1 drops it, both ranks). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live receipts — stage-1 store-only tier (server window 2026-09-01/02)Captured on the production pair (deb8 head / d9d5 worker) in the post-#77 server window. All blocks below are condensed/redacted excerpts of the raw transcripts (kept in our receipts mirror), not full verbatim pastes. One launcher defect was found live and fixed during the window (receipt 2a) — provenance per receipt:
1. knob=0 byte-identical serving — CAPTUREDControl boot, knobs off. The (The 2a. Launcher defect found live — fixed in
|
…NGES, 3 minors adopted) 1. Test harness: _env_of_worker only counts KEY=value tokens immediately following -e, so a drifted worker env assignment fails the env checks instead of passing on a stray token. 2. Comment/log accuracy: vLLM rejects OffloadingConnector under expandable_segments:True UNLESS the CuMem allocator is enabled (not on this deployment) - start.sh comment, launcher log line, D1b test comment. 3. (PR body edit, no code) blast-radius claim corrected + allocator-drop disclosure paragraph; see CODEX-087B5EA-REVIEW.md disposition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Follow-up: the owed adversarial review of The receipts comment above disclosed that Three minor findings. Findings 1–2 were adopted in
On the entangled +8.3% receipt-4 result: an isolation protocol is now written and adversarially reviewed — a three-arm boot design (control / allocator-dropped-without-store via a test-only scratch-branch hook / knob=1) with pre-declared bars. It is not a full factorial: the stock-allocator+store cell is unbootable on this deployment's non-CuMem path, so the design attributes conditional effects only (allocator effect with the store off; on-path store overhead under the dropped allocator — the background write path is priced separately since it is off the request path). It runs in the next server window; the outcome lands in this thread under the pre-declared bars, which include a not-reproduced outcome and a mixed/no-attribution outcome as well as the allocator-accounts and store-path-cost verdicts. |
Receipt 4 update: the +8.3% store overhead did not reproduce under the pre-registered isolation protocolReceipt 4 in this thread reported knob=1 cold prefill at ~26K tokens +8.3% vs control, failing the within-noise bar, with a caveat that it was entangled with the mandatory allocator change. We ran the pre-registered three-arm isolation protocol on the same pair (boot-isolated arms, execution order A→C→B, 8 cold reps per arm, fresh
Median paired deltas (per-index, bootstrap 95% CI, 10,000 percentile resamples): C−A +0.36 s [+0.21, +1.51]; C−B +0.49 s [+0.31, +1.81]; B−A −0.19 s [−0.30, +0.07]. Individual C−A deltas reached +3.55 s (arm C's first two reps carried boot-settle tails of 29.24/27.03 s); settled medians (reps 3–8): A 25.63 s, C 25.95 s. A′ drift check passes (|25.24 − 25.55| = 0.31 < T). The 3-rep decode probe was within its band in all arms. Verdict per the protocol's first-match ladder: R0 — the earlier +8.3% did not reproduce. This run does not establish whether the earlier excess was environmental, allocator-related, or store-related; it establishes that under quiescence and cold-guard gates the knob=1 on-path overhead sits within the pre-declared tolerance. Execution exceptions, disclosed: arm B's first boot was voided before any reps (warm-up never settled — the intermittent boot-settle failure noted earlier in this thread); the single contingency reboot supplied B's eight valid reps. Store activity during arm C wrote 136 files / ~2.80 GB per rank; the driver's post-run empty-root gate FAILED (root-owned files survived a non-sudo clear) — roots were cleared with sudo afterwards and now verify empty on both ranks (0 files head, 0 files worker). Production checkout and health verified restored ( Full receipts: Submitted by nood-co1 on behalf of Blockbrain Labs (https://x.com/blockbrain_labs, GitHub org blockbrain-ai). Measured on our 2× NVIDIA DGX Spark deployment. |
…-only # Conflicts: # .env.example # README.md # start.sh
|
Updated to current main (e25aa70) by merging main into the branch — merge commit 26ca3cd, plain push, no force-push; tested history preserved (83e4bd5…0431c66 incl. the disclosed 087b5ea alloc-conf fix remain ancestors). Conflicts: start.sh (7 hunks — kv-offload knob capture/config/bool-flag helper/validator block/-e lines vs main's indexer-workspace/spinwait equivalents, both sides kept; both rank-script apply blocks re-derived onto |
plotarmordev
added
triage/skip
Review follow-up —
|
Maintainer architecture decision — close the store-only stage, preserve its workWe are declining to adopt a store-only stage without a usable end-to-end hybrid restore path and are closing this PR on architecture/scope grounds, not as a duplicate or a defect-driven rejection. #58 is the experimental offload direction we will pursue; it is not qualified or approved to merge, and it does not replace the unique format/GC work here. We reviewed #100's closed-unmerged disposition: its group-0 restore is inert on this model's seven-group hybrid layout because the mamba groups have no restore source and its load-failure integration requires a single total group. The later single-group probe reached disk-job creation but did not establish successful worker restore. Thus #100 is not an available restore stage that makes adopting #99 alone deliver the intended benefit. Thank you @nood-co1 and Blockbrain Labs for the worker-local disk format, namespace/rank fencing, chunk headers and CRCs, durable boundary manifests, GC/verifier design, eligible-group scoping port, and unusually explicit receipts. The earlier +8.3% overhead did not reproduce under your predeclared isolation protocol; we are not reviving that result as a rejection reason or asking for the pending receipt cells. The source branch |
Stage-1 store-only disk KV tier: eligible-group scoping + worker-local store with chunk headers (
GLM53_KV_OFFLOAD)Status: knobs default OFF (control boot byte-identical). Host-side evidence below is complete; the live boot receipts (section ‘Receipts’) will be posted in this thread at the next server window — a measurement run currently owns the pair.
window after the #77 measurement pass. Codex gates: advisory
CODEX-OFFLOAD1-PLAN.md (DO-NOT-PROCEED v1 → all findings adopted/rebutted in
writing before build), adversarial diff review CODEX-OFFLOAD1-REVIEW.md
(DO-NOT-SHIP → all findings adopted/rebutted, fixes in 32f993e, confirm pass
appended there).**
Problem
A session whose KV has left the ~455K-token resident GPU envelope reprefills from
zero (174 s measured at 120K; 15–25 min estimated at 450K). The research plan
(PLAN-KV-OFFLOAD.md v2.1 + stage-0 receipts) designs a disk tier: park-to-disk on
store-cascade, restore on next touch. This PR is stage 1 only: store-only,
restore disabled by flag (the launcher refuses
GLM53_KV_OFFLOAD_RESTORE=1),no restore/read path — the tier writes; nothing reads it back yet. knob=0 is
byte-identical serving; knob=1 carries one process-wide side effect (the
allocator change described in the Launcher section below).
Three things stand between the image's shipped
kv_offloadstack (present —stage-0 receipt O4/R14) and a safe store on this deployment:
build_offloading_configasserts
tokens_per_block % tokens_per_hash == 0for every KV-cache group;the KpoolTailSpec scratch group (block 4,
prefix_cacheable=False) fails(4 % 64). Upstream PR [KV Offload] Scope offload group configs to prefix-cacheable KV cache groups vllm-project/vllm#54743 (ours) scopes the group configs
to prefix-cacheable groups; this overlay ports that scoping onto the image's
older tree.
UniformTypeKVCacheSpecswrappers (subclass ofKVCacheSpeconly): thescheduler's isinstance dispatch would crash on group 0's full-attention
assert and silently strip the mamba groups' alignment/CoW semantics. The
port adds a fail-closed single-type unwrap and reads cacheability through
the image's
participates_in_prefix_cachingname.TieringOffloadingSpec.get_manager()constructs every secondary tierscheduler-side against the scheduler node's staging memoryview; on
--nnodes 2rank 1's staging mmap lives on the other Spark, so the fs tierwould persist rank-0 bytes plus never-written garbage as the rank-1 half of
every file — corruption that only surfaces at restore time. The store path
must be worker-local: each rank writes its own bytes from its own
staging mmap to its own NVMe.
What this ships
overlay/patch_kv_offload_scope.py— eligible-group scoping (port of vllm#54743)OffloadingGroupConfiggainsgroup_idx(original boot index, required);build_offloading_configfilters to prefix-cacheable groups minus the forkpolicy exclusion of draft-model attention groups (
is_eagle_group, or exactSlidingWindowSpecunder an EAGLE speculative config) unlessGLM53_KV_OFFLOAD_DRAFTER=1; the divisibility assert still guards eligiblegroups.
group_statessized by the full group count; every pairingthrough
group_idx;GPULoadStoreSpec.group_sizes/block_indicesstayfull-length with zeros for ineligible groups (worker layout unchanged);
supports_partial_tailconservatively requires every group eligible.upstream predicate, g6 out by fork policy (min-exempt drafter; Per-KV-cache-group prefix-cache retention: sparse drafter, dense MLA/mamba #83
retention-0). Logged at boot.
overlay/patch_kv_offload_store_local.py— worker-local store + §4 chunk headersTransferJob.glm53_store_meta):ordered keys aligned 1:1 with the job's block order, plus boundary-manifest
candidates (cumulative chunk-hash chains).
synchronously at DMA completion (capture-then-write —
reset_cache()frees staging rows regardless of pending disk work, so async mmap reads
could tear; captured at
get_finishedbefore this rank's ack, or rightafter
worker.wait()on the flush/reset path), then writes the privatebuffers (512 MB budget) to
GLM53_KV_OFFLOAD_DIR/glm53kv_<model>_<ns>_r<rank>/<hhh>/<hh>_g<g>/<hash>.binon its local NVMe — real (unpadded) bytes only (the slot-share padding
never hits disk), fsync + tmp-rename + dir-fsync; acks are never deferred.
Rank = the initialized TP rank cross-checked against the connector config;
any disagreement disables the writer (never cross-rank mixing).
(fences model/vllm/build/TP/KV-dtype/hash-algo/
num_speculative_tokens/per-group real page sizes/KDA state-ABI/format version), original group
index, per-layer segment table (conv/temporal split with shape+dtype+stride
for KDA layers), boundary token index, payload CRC, header CRC. Any mismatch
at read ⇒ reject.
every cumulative full-attention chunk are durable AND header-verified
(namespace+hash+group identity; per-chunk
[hash, len, crc]recorded forfull groups too); a key that failed to write this boot can never be
manifested (failed-key ledger). Dedup likewise requires a verifying
header, else the file is rewritten.
GLM53_KV_OFFLOAD_KEEP_BOUNDARIES, default 2):a manifest is superseded once beyond the K most recent boundaries of EVERY
chain containing it (mamba payloads unlinked; full-attention chunks stay
dense — cumulative references keep shallower boundaries restorable).
reprefill; nothing can serve wrong bytes); ENOSPC ⇒ writer pauses; crash
between group writes ⇒ orphans, no live manifest;
overlay/kv_offload_store_gc.py(dry-run default) verifies headers, reports orphans, sweeps on request.
get_num_new_matched_tokensshort-circuits to 0 externalhits under
GLM53_KV_OFFLOAD_RESTORE=0— no loads, no promotions, noWAITING_FOR_REMOTE_KVS.
Launcher (
start.sh)Knobs validated fail-closed inside the pre-stop gate (bad knob or broken overlay
artifact exits 2 before
restartstops anything); knob=0 adds no serveargv, no mounts, no store dir — byte-identical serving. Knob=1 appends identical
--kv-transfer-configJSON to both ranks (TieringOffloadingSpec,cpu_bytes_to_use,blocks_per_chunk=1,offload_prompt_only=falsesodecoded tokens store too, no secondary tiers — see deviations), mounts
the store dir rw on both nodes, ships both overlays to both ranks, pins
their order (scope → store_local) in
GLM53_OVERLAY_ORDER, and self-checksboth patchers' injected sources (
--check-injected) inside the pre-stopgate. The store namespace is fenced by the recipe checkout SHA
(
GLM53_KV_OFFLOAD_BUILD_ID).When
GLM53_KV_OFFLOAD=1, the launcher also omitsPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truefrom both containers(
087b5ea, found live in the receipt window): vLLM rejects OffloadingConnectorunder expandable segments unless the CuMem allocator is enabled, which this
deployment does not use. knob=1 therefore runs both processes on PyTorch's
default native caching allocator — a process-wide change, not scoped to the
store writer. Expandable segments exists to reduce fragmentation OOM under
changing allocation sizes, so long-context workloads on knob=1 can in principle
OOM despite apparent free capacity where knob=0 would not (
GPU_MEM_UTILheadroom is not a fragmentation guarantee). The knob=0 path keeps the stock
entry and remains byte-identical. The +8.3% cold-prefill overhead in the
receipts thread is entangled with this allocator change; an isolation
experiment (allocator-only vs store+allocator arms) is protocolized for the
next server window.
Deviations (recorded, with reasons)
adopted): the scheduler-side
FileSystemTierManagerbyte path is exactly theR0 hazard; configuring-then-neutering it adds fake cascade bookkeeping and a
stock-namespace config.json for no benefit. The worker-local writer IS the
disk tier; the in-image fs tier remains available untouched for stage-2 read
work.
blocks_per_chunk=1uniformly (plan §4 wanted 16 for MLA): the image has oneglobal chunk size; all eligible groups share block 3584. g0 chunk files carry
27,163,136 B payloads (11 MLA + 11 indexer pages — the stage-0 correction
pricing the indexer KV rides in automatically).
(Codex finding 2, partially rebutted): a failed key is un-retried this boot
but can never be manifested; the failure bit lands with stage 2's T4 work,
where it becomes load-bearing.
Tests (host-side, all green; suite 33 passed)
tests/test_kv_offload_scope.pytests/test_kv_offload_store_local.pytests/test_launcher_kv_offload.pyReceipts (pending: server window)
To be captured in the post-#77 window; the PR stays unopened until every line
has a receipt — and thereafter until the maintainer-response hold lifts:
GLM53_KV_OFFLOAD=0; thevllm serveargv is byte-identical to the previous boot (no--kv-transfer-config), no connector init in the log, no store mounts;serving probes green.
[glm53-kv-offload-scope] eligible groups: [0, 2, 3, 4, 5] of 7on both ranks; TieringOffloadingManager created with0 secondary tiers;
[glm53-kv-offload-store] writer up: rank 0/2 …on deb8and
rank 1/2on d9d5 with per-rank base dirs; namespace JSON on bothSparks' local NVMe; driver version pinned in the capture.
under
…_r0/**_g{0,2,3,4,5}/on deb8 AND…_r1/**on d9d5 with payloadsizes 27,163,136 (g0) / 21,086,208 (g2,g3) / 18,743,296 (g4,g5); boundary
manifests for every complete 3584 boundary within keep-K; per-rank byte
totals within ±10% of the stage-0 §C model; R0 rank-routing receipt: GC
verifier reopen + checksum pass on BOTH ranks' stores across a container
restart (verifier-level, not a GPU restore — stage 1 has no restore).
knob=0 on the standard probe within the kit's measured noise band.
GLM53_KV_OFFLOAD_RESTORE=1refused pre-stop with zero docker/ssh calls (live run); GC dry-run flags a
deliberately truncated chunk and a manifest-less orphan; ENOSPC simulation
(small loopback fs) pauses the writer with serving unaffected.
Submitted by nood-co1 on behalf of Blockbrain Labs (https://x.com/blockbrain_labs, GitHub org blockbrain-ai). Measured on our 2× NVIDIA DGX Spark deployment.