NVFP4-CSF: expand_scales() for ahead-of-call scale expansion (canonical twin of #477) by voipmonitor · Pull Request #478 · local-inference-lab/b12x

voipmonitor · 2026-10-04T16:47:24Z

Expose fused_moe.expand_scales(experts) to expand prepared NVFP4-CSF scales before a bound call. bind(scales_expanded=True) reuses the expanded scratch. Expansion runs on the experts device and returns false for other weight formats. Callers must order scratch reuse, expansion, and execution across streams.

Retain required micro-kernel barrier resets and the stage-readable W4A16 path. No existing caller changes are required.

Validation: 92 composed scale-storage and MoE tests pass on SM120, including expansion with another current device, explicit stream ordering, CUDA graph replay with frozen resolution, poisoned scratch, and allocation stability. No full-model serving or overlap performance claim.

Retain exact FP4 nibbles and caller-owned scale buffers. Decode only routed experts and pair gate/up and down scales without adding streams or PDL. Preserve native weight planning and existing Kimi X4T behavior. Qualify the master port with 67 tests and real-layer graph replay. Include serving-composition timing evidence and explicit qualification limits.

Slice native E2M1 experts and compressed UE8M0 scales without requantization. Allow Kimi TP1/2/4/8/12/16 geometry and reuse the paired scale decoder. Qualified serving geometry is TP16; other ranks require separate model qualification. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Retain all paired-scale and TP12 route ABIs through startup planning and register their caches for deferred-program reclamation. Native weight declarations and kernel arithmetic are unchanged. Validation: 75 X4T tests; real Kimi TP16 shard with autotuning and eight poisoned graph replays at each planned size from 1 to 1024 tokens.

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Co-authored-by: OpenAI Codex <noreply@openai.com>

…cratch Preserve native NVFP4 arithmetic and load TP extents without expanding resident scale storage. Retain paired CuTe programs for both routing integer types and qualify exact bytes and poisoned graph replay. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Preserve X4T and LSC aliases, checkpoint bytes and kernel implementations. Document one-bit MXFP4 and four-bit NVFP4 offsets from a row base byte. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <vit@voipmonitor.org>

Require CSF schema and tensor names, preserve TP slices and FP32 calibration, and support Qwen3.8-Flash-Next geometry. Reorder resident offsets for contiguous decoding and fuse micro-kernel barrier initialization. GPU component validation: 79 tests passed.

Consume lazy projection sources and explicit geometry in B12X. Keep model names, manifests and file ownership in the companion vLLM loader. Preserve checkpoint bytes, scale layout, FP32 calibration and caller-owned scratch. Qualify with 88 component tests and byte parity for complete layers from Kimi, DeepSeek, GLM and Qwen. Co-authored-by: Codex <noreply@openai.com>

Accept rank-local CPU scale planes in prepare_weights and create resident scale batches inside its existing MXFP4/NVFP4 branches. Move checkpoint I/O and TP slicing to the paired vLLM integration. Preserve decoder kernels, calibration, scratch ownership and checkpoint bytes. Validation: 69 B12X tests; 56 vLLM tests; ten complete layer/rank comparisons with 132 byte-identical prepared tensors. Full-model serving was not rerun. Co-authored-by: Codex <noreply@openai.com>

Select cooperative shared-slot reconstruction or indexed plane expansion during typed preparation. Preserve native scale bytes, A4 arithmetic, N64 tails, caller-owned scratch and frozen replay. Correct the native route-broadcast lifetime and deterministic shared-activation indexing. Declare scale and reduction programs before preparation freezes kernel resolution. Validation: 408 component and native-oracle tests, two strict preparation declaration tests, and nine registry checks pass. Serving measurements and their performance limits are recorded separately.

Record matched GLM TP4 and Qwen TP2 MTP measurements, source and telemetry hashes, prepared memory use, and component correctness. Separate output throughput from speculative step-rate diagnostics and retain C1/Qwen residual costs. Describe the numeric scale-operand layout and independent native MoE invariant fixes. Timing remains research-only under the recorded NVML 0x400 hardware condition.

Pair ready operands before consumer synchronization, decode only consumed gate halves, and select partial-atom zero padding inside the shared reader. Preserve native MMA loads, checkpoint bytes and prepared memory sizes. Qualify 105 byte/expert graph tests and four online-preparation composition cases.

Record exact source and artifact identities, 105 correctness cases, racecheck and online-preparation composition. Publish generated-token and engine-step deltas separately, retain the TP2 C1 residual, and scope all timings to the observed clock-event mask.

…ression Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Packed W4A16 experts prepared from NVFP4-CSF checkpoints keep their E4M3 K16 scales as stage-readable compressed storage (nvfp4_csf_packed: 32-byte atom records with a folded value table). Each W4A16 pipeline stage rebuilds its scales in shared memory from that storage, so decode and small prefill calls no longer expand every routed expert's scales into scratch before the MoE. Calls planned above the stage-read token limit (W4A16_CSF_STAGE_MAX_TOKENS) still expand their routed experts into the shared W4A16 scale scratch first, and so does a call the opt-in A4 prefill path takes (it reads expanded scales; the check uses getattr so it holds whether or not that path is present). GLM-5.3-Flash stored QAD CSF checkpoint, TP2/DCP2 MTP3 on 2x RTX PRO 6000 Max-Q: decode C8 +10% (+9% per step) and C1 +1.6% per step versus the per-layer expansion, prefill unchanged; needle-checksum exactness unchanged (498/500/500 vs 498/492 salted runs). Canonical twin of b12x#471 (Karmic beta 9e2d984), on top of #459. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…461) The publisher rejects an empty models list, so this fragment as published in the beta (fixed there by #461) is the one a canonical merge must carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

expand_scales(experts) expands every expert's NVFP4-CSF scales into the experts' scratch on the current stream (decode_all, the full expansion long prefills already use), and bind(..., scales_expanded=True) skips the per-call expansion. An integration whose layers share one scale scratch can expand the next layer on a side stream while that layer's attention runs, instead of serially before its MoE, where the expansion took 4% of a GLM-5.3 TP2 A4 prefill. Calls of the micro kernel, whose barrier reset is fused into the expansion, ignore the flag and expand as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit dfffeac)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a23ecef)

coderabbitai · 2026-10-04T16:47:31Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: local-inference-lab/b12x/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 52a0fe74-0464-4be9-9553-d9c7de4cf485

📥 Commits

Reviewing files that changed from the base of the PR and between 3a4ad73 and 9bac0f6.

📒 Files selected for processing (80)

.lil/changes/b12x-450-csf-canonical.json
.lil/changes/b12x-450-csf-mxfp8.json
.lil/changes/b12x-450-csf-preparation.json
.lil/changes/b12x-450-csf-tensor-sources.json
.lil/changes/b12x-450-moe-routing.json
.lil/changes/b12x-450-nvfp4-csf-cooperative.json
.lil/changes/b12x-csf-indexed-vectors.json
.lil/changes/b12x-csf-operand-reconstruction.json
.lil/changes/b12x-kimi-x4t-prepared-scales.json
.lil/changes/b12x-nvfp4-csf-expand-scales.json
.lil/changes/b12x-w4a16-csf-stage-scales.json
b12x/_lib/intrinsics.py
b12x/_lib/quant/csf_routing.py
b12x/_lib/quant/mxfp4_csf.py
b12x/_lib/quant/nvfp4_csf.py
b12x/_lib/quant/nvfp4_csf_inline.py
b12x/_lib/quant/nvfp4_csf_packed.py
b12x/_lib/quant/x4t_packed_scales.py
b12x/_lib/quant/x4t_scales.py
b12x/gemm/_shared/mxfp8_bmm.py
b12x/moe/_shared/execution.py
b12x/moe/_shared/kernels/dynamic.py
b12x/moe/_shared/kernels/silu.py
b12x/moe/_shared/kernels/w4a16/kernel.py
b12x/moe/_shared/kernels/w4a16/prepare.py
b12x/moe/fused_moe/__init__.py
b12x/moe/fused_moe/_impl.py
b12x/moe/fused_moe/_preparation.py
b12x/moe/fused_moe/_tuning.py
b12x/moe/fused_moe/api.py
b12x/moe/fused_moe/planning.py
b12x/moe/fused_moe/weights.py
benchmarks/benchmark_w4a16_csf_stage_scales.py
docs/lossless-scale-compression.md
tests/moe/test_mxfp4_csf.py
tests/moe/test_nvfp4_csf.py
tests/moe/test_nvfp4_csf_dispatch.py
tests/moe/test_w4a8_migration_corpus.py
tests/preparation/test_tuning_predicates.py
tests/preparation/test_x4t_moe_programs.py
tests/quantization/test_csf_routing.py
tests/quantization/test_mxfp4_csf.py
tests/quantization/test_nvfp4_csf.py
tests/quantization/test_nvfp4_csf_inline.py
tests/quantization/test_nvfp4_csf_packed.py
tests/quantization/test_x4t_packed_scales.py
validation/csf/README.md
validation/csf/compare_tensor_loaders.py
validation/csf/dynamic-moe-correctness.md
validation/csf/mxfp8-serving.md
validation/csf/nvfp4-scale-operands.md
validation/csf/nvfp4-serving-evidence.json
validation/csf/nvfp4-serving.md
validation/csf/preparation-evidence/b12x-api-baseline-lint.json
validation/csf/preparation-evidence/b12x-api-lint.json
validation/csf/preparation-evidence/b12x-prepare-tests.txt
validation/csf/preparation-evidence/ds-prepared-parity.json
validation/csf/preparation-evidence/glm-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp12-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp16-prepared-parity.json
validation/csf/preparation-evidence/qualification.json
validation/csf/preparation-evidence/qwen-prepared-parity.json
validation/csf/preparation-evidence/vllm-prepare-hooks.txt
validation/csf/preparation-evidence/vllm-prepare-mypy.txt
validation/csf/preparation-evidence/vllm-prepare-tests.txt
validation/csf/qwen-serving-evidence.json
validation/csf/qwen-serving.md
validation/csf/tensor-source-evidence/README.md
validation/csf/tensor-source-evidence/b12x-tests.txt
validation/csf/tensor-source-evidence/capture_layer_parity.py.txt
validation/csf/tensor-source-evidence/ds-layer-parity.json
validation/csf/tensor-source-evidence/glm-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp12-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp16-layer-parity.json
validation/csf/tensor-source-evidence/portable-tool-qwen.json
validation/csf/tensor-source-evidence/qualification.json
validation/csf/tensor-source-evidence/qwen-layer-parity.json
validation/csf/tensor-source-evidence/vllm-mypy.txt
validation/x4t/serving.json
validation/x4t/validate_layer.py

 _________________________________
< Winner winner, bug-free dinner. >
 ---------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…006T052218Z-2bca98-459

# Conflicts: # tests/moe/test_nvfp4_csf.py

…frozen-20261006T052218Z-2bca98-475 # Conflicts: # b12x/moe/fused_moe/_preparation.py # b12x/moe/fused_moe/_tuning.py

…006T052218Z-2bca98-475 # Conflicts: # tests/moe/test_nvfp4_csf.py

…rozen-20261006T052218Z-2bca98-478 # Conflicts: # b12x/moe/fused_moe/_impl.py # tests/moe/test_nvfp4_csf.py

voipmonitor and others added 23 commits September 29, 2026 22:54

Merge remote-tracking branch 'canonical/master' into review/kimi-x4t-…

c62107a

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Declare Kimi compressed-scale preparation for runtime releases

552da21

Co-authored-by: OpenAI Codex <noreply@openai.com>

Preserve FP32 NVFP4 calibration under BF16 model loading

8cb43b9

Preserve native MXFP8 expert execution for compressed MXFP4 scales

7dc9174

Vectorize indexed NVFP4 scale reconstruction and exception lookup

e646d0d

Qualify indexed scale reconstruction with Qwen TP1 and TP2 serving

566e288

docs: qualify CSF operand reconstruction without load-time scale comp…

993eac6

…ression Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

release: fragment for expand_scales (#477)

9bac0f6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a23ecef)

lukealonso added 2 commits October 5, 2026 23:04

Merge master into NVFP4-CSF reconstruction

5bd0dc2

fix(moe): retain staged-scale preparation contracts

1170054

lukealonso added 10 commits October 5, 2026 23:55

Merge remote-tracking branch 'origin/master' into review/frozen-20261…

d05502e

…006T052218Z-2bca98-459

Merge remote-tracking branch 'origin/master' into review/frozen-20261…

826eb42

…006T052218Z-2bca98-459

Merge commit '11700540' into review/frozen-20261006T052218Z-2bca98-478

ef0e265

# Conflicts: # tests/moe/test_nvfp4_csf.py

Merge branch 'review/frozen-20261006T052218Z-2bca98-459' into review/…

65b19ba

…frozen-20261006T052218Z-2bca98-475 # Conflicts: # b12x/moe/fused_moe/_preparation.py # b12x/moe/fused_moe/_tuning.py

fix(moe): order prefetched scale expansion and qualify graph replay

cefd362

Merge commit '65b19ba2' into review/frozen-20261006T052218Z-2bca98-478

fb61c86

Merge remote-tracking branch 'origin/master' into review/frozen-20261…

064c655

…006T052218Z-2bca98-475 # Conflicts: # tests/moe/test_nvfp4_csf.py

test(moe): cover scale expansion with another current device

5661a44

Merge commit '064c655c' into review/frozen-20261006T052218Z-2bca98-478

7b32dd6

Merge commit '3a4ad73a669dcf88641d038baa33bb79fd38c4c9' into review/f…

a25b7cd

…rozen-20261006T052218Z-2bca98-478 # Conflicts: # b12x/moe/fused_moe/_impl.py # tests/moe/test_nvfp4_csf.py

lukealonso changed the base branch from perf/w4a16-csf-stage-scales-canonical to master October 6, 2026 07:12

lukealonso merged commit c99e68d into master Oct 6, 2026

Navigation Menu

Repository navigation

NVFP4-CSF: expand_scales() for ahead-of-call scale expansion (canonical twin of #477) - #478

NVFP4-CSF: expand_scales() for ahead-of-call scale expansion (canonical twin of #477)#478
lukealonso merged 35 commits into
masterfrom
perf/csf-expand-scales-canonical

voipmonitor commented Oct 4, 2026 •

edited by lukealonso

Loading

Uh oh!

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

voipmonitor commented Oct 4, 2026 • edited by lukealonso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

voipmonitor commented Oct 4, 2026 •

edited by lukealonso

Loading

coderabbitai Bot commented Oct 4, 2026 •

edited

Loading