[faresobeid]

Adds sparse deltas to the filesystem weight broadcast. Between anchors, the trainer publishes only the weight elements that changed in the optimizer step, in the engine's weight format (bf16, or fp8 for engines that quantize online to fp8_per_block). Engines patch their weights in place and verify every load against trainer checksums.

[weight_broadcast]
type = "filesystem"

[weight_broadcast.delta]  # set => SparseFileSystemWeightSender; unset => full checkpoints as today
interval = 50  # full checkpoint (anchor) every N steps, for engines that join or fall off the delta chain

How it works

Trainer: deltas from the optimizer step. Just before the optimizer writes a parameter chunk, the local shard still holds the previous broadcast's weights, so the delta is taken from the step itself and no copy of the weights is kept.

• GPU optimizers (offload = "none" / "optimizer"): AdamW / SignSGD step up to 64M local elements at a time and call update_hook around each chunk (ChunkedUpdates). Muon steps its own batches of matrices as chunks (the same on every rank, since they run collectives); per-group step counts still advance once per step. offload = "full": FullCPUOffloadOptimizer.weight_hook replaces the H2D copy of the new values.
• The hook copies the chunk's old wire bits into a reused GPU slot. Right after the update, Triton kernels compare them with the new bits in place and write the changed (index, value) pairs and per-tensor checksums straight into pinned host memory, without a host sync; a thread turns them into delta parts while the step runs. fp8 and fp32 tensors take a diff thread instead.
• Anchor slices: GPU optimizers write them during the step; with full offload they are written from the live weights after the delta is committed (writing them in the step stalls the gradient offload).
• After the step, a position-weighted checksum of the live weights checks that every optimizer tensor holds what the step wrote; a weight changed outside the optimizer step raises.
• A tied weight's deltas go out once, under its first name; anchors and full checkpoints carry every name.
• Tensors that are not optimizer parameters (buffers, frozen parameters, fp8 tensors whose 128-row blocks straddle ranks) are diffed against a small copy taken at the last full checkpoint.
• Broadcasts that do not follow an optimizer step this way publish a full checkpoint: a resumed run's first broadcast and SFT online-eval broadcasts. Step 0 is the checkpoint the engines booted from.

Format. The prime->HF conversion is replayed once on meta tensors, so each changed element maps to an HF tensor and index. Each rank writes zstd part files (index gaps + zigzag value differences, ~1.5 B per change) with a position-weighted checksum for every HF tensor. Anchors are per-rank slices of a full checkpoint, written without a gather; full checkpoints carry per-tensor checksums too.

FP8 engines (online fp8_per_block, incl. DeepGEMM ue8m0). Every engine worker (one per DP, PP and TP rank) reports which tensors it holds in fp8 and their scale format at every full broadcast; the trainer clears the last run's reports and takes them at its first broadcast. The trainer quantizes old and new weights with vLLM's quantizer (compiled once with AOTInductor at the startup broadcast, bit-identical) and diffs the fp8 bytes; changed block scales travel in the delta as {name}_scale_inv. MLA absorbed weights are recomputed after each delta.

Engine (FileSystemWeightUpdateWorker).

• /prefetch_weights reads and decompresses this worker's parts while the engine keeps serving, into a reused pinned buffer. The orchestrator does this in WeightReceiver.prepare, before it stops train dispatch for the update, so rollouts keep being dispatched meanwhile. In the pause the worker applies the delta in place on the GPU and verifies every tensor it holds.
• The checkpoint->param map is recorded once by running vLLM's own load_weights on placeholders (name mapping, qkv/gate_up/expert packing, TP/EP slicing; no per-model tables).
• A fresh, relaunched or failed engine loads the latest complete anchor, reading only the rows it holds, verifies it from its params as loaded (so a mapping or write bug in the load fails the check), and replays the deltas since. Before probing a step for a delta, the worker revalidates the step dir, so a stale NFS lookup cache can't turn a delta step into a full load. Tensors vLLM casts on load (e.g. an fp32 norm from a bf16 checkpoint) load by value with the same cast; a delta that changes one raises (none do today: indexers and RL routers are frozen).
• The trainer removes an older delta chain only once a newer anchor was complete one broadcast earlier, so an engine replaying the older chain never loses it mid-replay.
• The index map follows the engine's kernel layout. At the first update, while the engine still holds its boot weights, every recorded write is checked byte for byte against the boot checkpoint; a write whose block the engine holds elsewhere in the same param moves there, and a block found nowhere raises. Kernel processing after loading can move blocks: FlashInfer CUTLASS (the unquantized MoE backend on SM120) swaps the gate and up halves of fused expert weights, so without this every MoE engine on SM120 failed its first delta and the replay that followed loaded swapped experts while still passing verification (greedy generation changed after an anchor load). Measured on 2x RTX PRO 6000 with mini-glm-moe: 184 writes moved per worker with EP, 368 with TP only; afterwards 0 mismatches and identical greedy generation after an anchor load. H200 picks Triton for unquantized MoE and is unaffected.
• One plan per TP group. TP rank 0 plans each update (checkpoint to load, deltas to apply, full or incremental verify) and broadcasts it; the other workers follow it. Workers can no longer split between apply, replay and anchor load (an anchor completing mid-update, or a failure on one worker) and issue different collectives. A failure on any worker fails the update on all of them (gloo flag), so they all reload and replay the same way next time; a planning failure on rank 0 is broadcast too.
• Verification runs one fixed-size check over every checkpoint tensor of the run, so every worker issues the same reductions whatever it holds. A tensor is checked against multiplicity x trainer checksum when its holders cover it evenly: TP slices once, KV heads replicated over TP workers (TP > num KV heads) once per replica. Before, replicated KV heads were summed once per replica and never matched.
• Checksums move with the changes: apply_delta updates each tensor's checksum by position x (new - old) of its changed elements (O(changes) instead of a pass over every param); steps that carry an anchor recompute them in full. The static half of the TP check (positions, counts, how evenly the TP group covers each tensor, and three of its four reductions) is computed once per index map.
• One fused Triton kernel applies a delta (inference/vllm/worker/sparse_apply.py): each change finds its tensor by binary search over the delta's entries, maps through device tables built once per index map, adds its bits in place (indices are unique, so no sort or atomics) and returns its running-checksum contribution. It replaces the per-chunk tables, searchsorted, argsort and index_put of the torch path, which stays for CPU and as the reference. It is compiled when the index map is built, with do_not_specialize on its sizes and pointers so no delta recompiles it inside the pause (we saw 0.7-2 s apply spikes from that).
• Writes can carry row/column offset tables instead of strides (element (r, c) at offset + rows[r] + cols[c]), through loads, deltas and checksums: what kernel layouts that shuffle rows or swizzle block scales need (Blackwell MXFP8/NVFP4 backends; used by the MXFP8 follow-up PR).
• The apply log line splits GPU time into copies, index decode and patch (CUDA events), next to the host loop.

Earlier results (before this round)

All runs on H200; every update verified against trainer checksums.

setup (GLM-4.5-Air, 1 trainer node, EP8 engines unless noted)	sparse	NCCL	NIXL + overlap
1 engine	23.1-23.7 s/step	28.9 s	23.4 s
4 engines	23.0 s	29.1 s
8 engines (64 inference GPUs)	24.7 s	30.9 s
one engine across 2 nodes (DP16+EP16)	23.7 s	28.7 s	24.1 s
1 prefill + 1 decode, router replay	21.6 s	30.0 s	29.4 s
GLM-5.3 744B, 8 trainer nodes + one fp8 engine on 2 nodes	~24 s on transfer-bound steps	106.4 s

• Trainer broadcast 0.2-0.3 s and engine pause ~1 s on GLM-4.5-Air (NCCL 8.5-9 s); on GLM-5.3 the pause is 3-4 s (NCCL 86 s).
• Storage is not the bottleneck at 64 readers: per-worker reads stay at 10-11 s, the aggregate scales to ~27 GiB/s.
• Deltas: ~4% of elements per step with sign_sgd, 0.6-1.2% with AdamW; GLM-5.3 3.6-4.2 GiB per step with 1-6 fp8 scale changes, all in the delta.
• A fresh GLM-5.3 engine loads its 119 GiB slice of a 1.4 TB anchor in 128-180 s, then replays.
• Normal-step cost of diffing in the optimizer step (Qwen3-30B, same node vs a full-copy diff): equal for AdamW with offload = "optimizer" and with full offload; +0.34 s for AdamW "none".
• Trainer GPU memory: GLM-4.5-Air full offload peaks at 55.9-56.7 GiB on every rank vs 54.4-56.1 GiB (rank 0: 60.5) for NIXL + overlap on the same nodes; peak allocated is the same (49-50 GiB). Qwen3-30B AdamW offload = "optimizer": 82.4 GiB vs 89.5 GiB with full-copy diffs.
• Anchors are storage-bound: +10 s on Qwen3-30B, ~+21 s on GLM-4.5-Air (full offload: the next step waits for the writes), +26 s on GLM-5.3 (anchor written in the step). At anchor_interval = 50 that averages to ~0.2-0.5 s per step.
• SFT online evals: full-checkpoint broadcasts (27-30 s on Qwen3-30B), loaded and verified by engines.

Exactness. Every delta, anchor and full-checkpoint load passed the checksum check. A fresh engine joining at a non-anchor step verifies; fail-then-retry heals; a double-applied or off-by-one delta, and a corrupted anchor, fail the check. fp8 in-place patches are byte-identical to a full fp8 reload (Qwen3-30B, DeepSeek-V3 with MLA), including scale changes.

Per-op timings on the current head

H200, reverse-text, batch 128 × 16, max 512 completion tokens, 40 steps (GLM: 30), anchor every 20 steps (GLM: 15). Medians over steps > 3. Single node: 4 trainer + 4 inference GPUs (Qwen3-30B-A3B: 6 + 2, TP2). GLM-4.5-Air: 1 trainer node (EP8, full offload) + 1 inference node (DP8+EP). "Dispatch-blocking update" is the time train dispatch is stopped per step (pause + apply/verify + resume for sparse, pause + transfer + resume for NCCL). Every sparse delta was applied and checksum-verified on every engine worker (0 mismatches); anchors were written and verified at write time.

model / setup	transport	step s (median)	trainer broadcast s	dispatch-blocking engine update s (pause / apply+verify or transfer / resume)	delta MiB/step	trainer encode / commit s	engine read+decode while serving s	anchor extra s (once per interval)	reward first5 → last5	mismatch KL
Qwen3-0.6B AdamW	sparse	2.96	0.13	0.18 (0.03 / 0.12 / 0.01)	5.9	0.01 / 0.08	0.13	n/m	0.02 → 0.24	0.0008
Qwen3-0.6B AdamW	NCCL	2.75	0.39	0.33 (0.03 / 0.27 / 0.01)	-	- / -	-	-	0.03 → 0.24	0.0010
Qwen3-0.6B Muon	sparse	2.72	0.13	0.18 (0.03 / 0.13 / 0.01)	20.9	0.02 / 0.11	0.17	n/m	0.03 → 0.28	0.0041
Qwen3-0.6B Muon	NCCL	2.31	0.44	0.33 (0.03 / 0.27 / 0.01)	-	- / -	-	-	0.04 → 0.28	0.0057
Qwen3-8B AdamW	sparse	6.38	0.13	0.23 (0.04 / 0.19 / 0.01)	85.1	0.04 / 0.28	0.43	8.11	0.00 → 0.64	0.0013
Qwen3-8B AdamW	NCCL	6.78	0.93	0.40 (0.04 / 0.37 / 0.01)	-	- / -	-	-	0.00 → 0.58	0.0015
Qwen3-8B Muon	sparse	5.25	0.13	1.10 (0.04 / 0.27 / 0.01)	245.2	0.02 / 0.16	0.67	10.23	0.00 → 0.66	0.0076
Qwen3-30B-A3B AdamW (opt. offload)	sparse	6.16	0.15	0.50 (0.03 / 0.46 / 0.00)	284.7	0.12 / 0.39	0.76	14.08	0.21 → 0.60	0.0220
Qwen3-30B-A3B AdamW (opt. offload)	NCCL	11.77	5.61	4.10 (0.03 / 4.02 / 0.01)	-	- / -	-	-	0.21 → 0.60	0.0289
GLM-4.5-Air sign_sgd (full offload), 2 nodes	sparse	20.27	0.17	0.96 (0.06 / 0.83 / 0.02)	4672.7	0.14 / 1.14	5.12	17.10	0.46 → 0.81	0.0130
GLM-4.5-Air sign_sgd (full offload), 2 nodes	NCCL	22.33	9.45	8.50 (0.04 / 8.49 / 0.03)	-	- / -	-	-	0.42 → 0.83	0.0151
Qwen3-30B-A3B AdamW (opt. offload)	filesystem	157.05	151.13	84.86 (0.03 / 84.82 / 0.00)	-	- / -	-	-	0.20 → 0.46	0.0204

• Sparse is 1.9× faster per step on Qwen3-30B-A3B (6.2 vs 11.8 s) and 9% faster on GLM-4.5-Air (20.3 vs 22.3 s): the trainer broadcast drops from 5.6-9.4 s to 0.15-0.17 s, and the dispatch-blocking update from 4.1-8.5 s to 0.5-1.0 s. The delta is read and decoded while the engines serve (0.8-5.1 s), off the critical path. Rewards and mismatch KL match NCCL.
• Qwen3-8B is trainer-bound for both transports (step 6.4 vs 6.8 s, the difference is the 0.8 s broadcast).
• Qwen3-0.6B is slower with sparse (3.0 vs 2.75 s; Muon 2.7 vs 2.3 s): the run becomes rollout-bound (the trainer waits ~0.3 s/step for batches), and the engine-side read/decode overlapping generation costs some decode speed (inter-token latency 3.1 vs 2.7 ms with Muon) on this tiny, CPU-bound model. At 8B+ it does not show (equal latency). Use NCCL for tiny models on one node.
• GLM-4.5-Air full offload with sign_sgd (~4% of elements change, 4.7 GiB delta, 828M changes per step): diffing in the optimizer step adds ~6 s to the overlapped final backward (forward+backward 19.1 vs 12.8 s). NCCL's 9.4 s broadcast costs more, so sparse is still ahead. AdamW changes 0.7-1.2% of elements.
• Anchor cost (once per interval): +8 s (Qwen3-8B), +14 s (Qwen3-30B), +17 s (GLM-4.5-Air); not measurable on Qwen3-0.6B. At interval = 50 that is 0.2-0.35 s per step.
• Plain filesystem broadcast for reference (Qwen3-30B-A3B): 151 s per broadcast, 157 s/step.
• Qwen3-8B Muon ran before the dispatch fix (its 1.10 s window includes the read). With the fix the window is 0.18-0.24 s (Qwen3-0.6B/8B).

Validation

On vLLM 0.33 (chore/vllm-0.33 merged in, so until #4365 lands the diff against main also shows the vLLM bump):

• CPU: pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/utils tests/unit/transports tests/unit/inference tests/unit/train/test_chunked_updates.py -m "not gpu": 552 passed (551 before the plan/checksum change, plus test_running_checksums_follow_deltas); ruff clean.

• GPU (RTX PRO 6000, vLLM 0.33): a TP=4 engine of a Qwen3-0.6B variant with 2 KV heads (each replicated on 2 TP workers) loading a trainer-format anchor through update_weights_from_path: before this change every layer's k_proj/v_proj failed verification (diverged from the trainer); now the anchor loads and verifies, and generation is unchanged.

• Qwen3-4B (TP=2 engine, 2 trainer GPUs, AdamW, 12 steps, anchor every 5; 130-190M changes per update, 3-5% of elements), seconds per update on one engine worker, RTX PRO 6000:

  head	apply	verify	apply + verify
  before (7eb55c659)	0.38-2.16	1.63-4.20	~2.2-6.3
  one plan per TP group + running checksums, torch apply (0ea767d4f)	0.94-10.1	0.01 (first 0.26)	~1-10
  + fused kernel (this head)	0.09-1.39	0.00-0.07 (first 0.35)	~0.1-1.4

  Every update on the fused head was applied and verified (24/24, 0 mismatches) and the run trained normally (reward 0.25 -> 0.40). 0ea767d4f alone is slower on GPU (the torch path's running-checksum gathers); it ships with the fused kernel. The remaining apply spread (0.1-1.4 s for the same work) is being traced with the new per-phase log.

• Kernel microbenchmark (1 GiB of bf16 params, RTX PRO 6000): apply torch 10.1 ms / fused 2.3 ms at 5M changes, 25.1 / 2.7 ms at 21M; full checksum 25 ms vs running 0.1 ms.

• GPU unit tests: tests/unit/inference/test_checkpoint_index_map.py (fused apply bit-identical to the torch path, mapped writes included) pass.

Earlier (vLLM 0.32):

• Up to date with main (plain merge). CPU on the head: pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/utils tests/unit/transports tests/unit/inference tests/unit/train/test_chunked_updates.py -m "not gpu" 531 passed; ruff check, ruff format --check, uv lock --check clean.
• GPU: the runs above. A tied model (Qwen3-0.6B): its anchors carry lm_head.weight byte-identical to model.embed_tokens.weight.

Notes

• Not supported: LoRA (raises); fp8 tensors whose columns are not a multiple of 128; engines that transform weights after loading other than online fp8_per_block; conversions that copy or transform values.
• Trainer and engines must use the same vLLM/torch for fp8 deltas (the quantizer is a compiled kernel).
• Every broadcast must follow an optimizer step to be a delta; anything else publishes a full checkpoint.
• In-place patching keeps param storage, so CUDA graphs captured at boot stay valid.
• Receiver polls revalidate NFS directories (as in fix(weights): revalidate NFS lookups when polling for handshake markers #4081).

---

Note

High Risk
Changes the RL weight-sync path (sparse deltas, in-place patches, cross-rank verification) alongside a major vLLM/torch upgrade; mis-sync or checksum bugs would silently or loudly break inference/trainer alignment.

Overview
Adds sparse filesystem weight broadcast: with [weight_broadcast.delta], the trainer publishes only changed weight elements between periodic full anchors, and inference workers prefetch, patch in place, and checksum-verify updates instead of reloading full checkpoints.

The trainer diffs each optimizer chunk via a new ChunkedUpdates mixin and Triton sparse_kernels, overlapping encode with the step; the orchestrator calls prepare / /prefetch_weights while engines still serve, then applies deltas through an expanded FileSystemWeightUpdateWorker (CheckpointIndexMap, fused sparse_apply kernel, anchor replay, TP-group planning).

Also bumps the GPU stack to vLLM 0.33 / torch 2.14 / NIXL 1.5, drops the standalone flash-mla wheel (FlashMLA from vLLM), removes obsolete DeepSeek V4 vLLM monkey patches, and adjusts routed-experts / LoRA / token serving patches for vLLM 0.33 behavior.

^{Reviewed by Cursor Bugbot for commit 1de91f1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[wiz-prime-intellect]

Wiz Scan Summary

Scanner	Findings
Vulnerabilities	-
Sensitive Data	-
Secrets	-
IaC Misconfigurations	-
SAST Findings	7
Software Management Findings	-
Total	7

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1de91f1. Configure here.}

[cursor]

Boot read requires every recorded name

Medium Severity

The first sparse update now always reads every recorded checkpoint name from the engine boot dir via _read_rows. A name the loader recorded from the trainer (for example a tied lm_head.weight omitted from the Hugging Face safetensors) is missing from checkpoint_index and raises, so the update never starts. Before this, the boot index was only used for fp8 layout checks.

Additional Locations (1)

• src/prime_rl/inference/vllm/worker/filesystem.py#L476-L478

 

^{Reviewed by Cursor Bugbot for commit 1de91f1. Configure here.}