Prepare FP4-CSF scales and repair expert batch decoding by voipmonitor · Pull Request #450 · local-inference-lab/b12x

voipmonitor · 2026-10-01T00:59:29Z

Prepare MXFP4-CSF and NVFP4-CSF expert scales through the existing MoE weight API. Caller-owned scale scratch supports serialized layer reuse, while retained decoders and inline operands support frozen CUDA graph execution. Correct shared routing synchronization and deterministic activation gathers.

CsfScalePlanes, Mxfp4CsfWeights, and Nvfp4CsfWeights accept numeric scale planes. Mxfp4CsfWeights replaces X4TWeights; checkpoint reading belongs to vLLM. Query/config/candidate versions are 16/9/18. Preparation rejects overlapping scale outputs and projections on different devices.

Validation: 376 composed CSF, W4A8, preparation and graph tests passed on SM120; six additional overlap/cross-device rejection cases passed on two GPUs. Changed-code undefined-name lint and diff checks pass. Full-model serving and throughput were not rerun locally; historical reports retain their source identities and limits.

Retain exact FP4 nibbles and caller-owned scale buffers. Decode only routed experts and pair gate/up and down scales without adding streams or PDL. Preserve native weight planning and existing Kimi X4T behavior. Qualify the master port with 67 tests and real-layer graph replay. Include serving-composition timing evidence and explicit qualification limits.

Slice native E2M1 experts and compressed UE8M0 scales without requantization. Allow Kimi TP1/2/4/8/12/16 geometry and reuse the paired scale decoder. Qualified serving geometry is TP16; other ranks require separate model qualification. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Retain all paired-scale and TP12 route ABIs through startup planning and register their caches for deferred-program reclamation. Native weight declarations and kernel arithmetic are unchanged. Validation: 75 X4T tests; real Kimi TP16 shard with autotuning and eight poisoned graph replays at each planned size from 1 to 1024 tokens.

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Co-authored-by: OpenAI Codex <noreply@openai.com>

…cratch Preserve native NVFP4 arithmetic and load TP extents without expanding resident scale storage. Retain paired CuTe programs for both routing integer types and qualify exact bytes and poisoned graph replay. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Preserve X4T and LSC aliases, checkpoint bytes and kernel implementations. Document one-bit MXFP4 and four-bit NVFP4 offsets from a row base byte. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <vit@voipmonitor.org>

coderabbitai · 2026-10-01T00:59:37Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: local-inference-lab/b12x/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 66034aba-7d70-4c53-bc88-431400bc51fa

📥 Commits

Reviewing files that changed from the base of the PR and between 3e67fe1 and f838b9b.

📒 Files selected for processing (69)

.lil/changes/b12x-450-csf-canonical.json
.lil/changes/b12x-450-csf-mxfp8.json
.lil/changes/b12x-450-csf-preparation.json
.lil/changes/b12x-450-csf-tensor-sources.json
.lil/changes/b12x-450-moe-routing.json
.lil/changes/b12x-450-nvfp4-csf-cooperative.json
.lil/changes/b12x-kimi-x4t-prepared-scales.json
b12x/_lib/quant/csf_routing.py
b12x/_lib/quant/mxfp4_csf.py
b12x/_lib/quant/nvfp4_csf.py
b12x/_lib/quant/nvfp4_csf_inline.py
b12x/_lib/quant/x4t_packed_scales.py
b12x/_lib/quant/x4t_scales.py
b12x/moe/_shared/execution.py
b12x/moe/_shared/kernels/dynamic.py
b12x/moe/_shared/kernels/silu.py
b12x/moe/_shared/kernels/w4a16/kernel.py
b12x/moe/_shared/kernels/w4a16/prepare.py
b12x/moe/fused_moe/__init__.py
b12x/moe/fused_moe/_impl.py
b12x/moe/fused_moe/_preparation.py
b12x/moe/fused_moe/_tuning.py
b12x/moe/fused_moe/api.py
b12x/moe/fused_moe/planning.py
b12x/moe/fused_moe/weights.py
docs/lossless-scale-compression.md
tests/moe/test_mxfp4_csf.py
tests/moe/test_nvfp4_csf.py
tests/moe/test_nvfp4_csf_dispatch.py
tests/moe/test_w4a8_migration_corpus.py
tests/preparation/test_tuning_predicates.py
tests/preparation/test_x4t_moe_programs.py
tests/quantization/test_csf_routing.py
tests/quantization/test_mxfp4_csf.py
tests/quantization/test_nvfp4_csf.py
tests/quantization/test_nvfp4_csf_inline.py
tests/quantization/test_x4t_packed_scales.py
validation/csf/README.md
validation/csf/compare_tensor_loaders.py
validation/csf/dynamic-moe-correctness.md
validation/csf/mxfp8-serving.md
validation/csf/nvfp4-scale-operands.md
validation/csf/nvfp4-serving-evidence.json
validation/csf/nvfp4-serving.md
validation/csf/preparation-evidence/b12x-api-baseline-lint.json
validation/csf/preparation-evidence/b12x-api-lint.json
validation/csf/preparation-evidence/b12x-prepare-tests.txt
validation/csf/preparation-evidence/ds-prepared-parity.json
validation/csf/preparation-evidence/glm-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp12-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp16-prepared-parity.json
validation/csf/preparation-evidence/qualification.json
validation/csf/preparation-evidence/qwen-prepared-parity.json
validation/csf/preparation-evidence/vllm-prepare-hooks.txt
validation/csf/preparation-evidence/vllm-prepare-mypy.txt
validation/csf/preparation-evidence/vllm-prepare-tests.txt
validation/csf/tensor-source-evidence/README.md
validation/csf/tensor-source-evidence/b12x-tests.txt
validation/csf/tensor-source-evidence/capture_layer_parity.py.txt
validation/csf/tensor-source-evidence/ds-layer-parity.json
validation/csf/tensor-source-evidence/glm-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp12-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp16-layer-parity.json
validation/csf/tensor-source-evidence/portable-tool-qwen.json
validation/csf/tensor-source-evidence/qualification.json
validation/csf/tensor-source-evidence/qwen-layer-parity.json
validation/csf/tensor-source-evidence/vllm-mypy.txt
validation/x4t/serving.json
validation/x4t/validate_layer.py

 ____________________________________________________
< Veni, Vidi, Verificavi. I came, I saw, I verified. >
 ----------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Require CSF schema and tensor names, preserve TP slices and FP32 calibration, and support Qwen3.8-Flash-Next geometry. Reorder resident offsets for contiguous decoding and fuse micro-kernel barrier initialization. GPU component validation: 79 tests passed.

Consume lazy projection sources and explicit geometry in B12X. Keep model names, manifests and file ownership in the companion vLLM loader. Preserve checkpoint bytes, scale layout, FP32 calibration and caller-owned scratch. Qualify with 88 component tests and byte parity for complete layers from Kimi, DeepSeek, GLM and Qwen. Co-authored-by: Codex <noreply@openai.com>

Accept rank-local CPU scale planes in prepare_weights and create resident scale batches inside its existing MXFP4/NVFP4 branches. Move checkpoint I/O and TP slicing to the paired vLLM integration. Preserve decoder kernels, calibration, scratch ownership and checkpoint bytes. Validation: 69 B12X tests; 56 vLLM tests; ten complete layer/rank comparisons with 132 byte-identical prepared tensors. Full-model serving was not rerun. Co-authored-by: Codex <noreply@openai.com>

Select cooperative shared-slot reconstruction or indexed plane expansion during typed preparation. Preserve native scale bytes, A4 arithmetic, N64 tails, caller-owned scratch and frozen replay. Correct the native route-broadcast lifetime and deterministic shared-activation indexing. Declare scale and reduction programs before preparation freezes kernel resolution. Validation: 408 component and native-oracle tests, two strict preparation declaration tests, and nine registry checks pass. Serving measurements and their performance limits are recorded separately.

Record matched GLM TP4 and Qwen TP2 MTP measurements, source and telemetry hashes, prepared memory use, and component correctness. Separate output throughput from speculative step-rate diagnostics and retain C1/Qwen residual costs. Describe the numeric scale-operand layout and independent native MoE invariant fixes. Timing remains research-only under the recorded NVML 0x400 hardware condition.

…461) The publisher rejects an empty models list, so this fragment as published in the beta (fixed there by #461) is the one a canonical merge must carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

voipmonitor and others added 8 commits September 29, 2026 22:54

Merge remote-tracking branch 'canonical/master' into review/kimi-x4t-…

c62107a

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Declare Kimi compressed-scale preparation for runtime releases

552da21

Co-authored-by: OpenAI Codex <noreply@openai.com>

Preserve FP32 NVFP4 calibration under BF16 model loading

8cb43b9

voipmonitor mentioned this pull request Oct 1, 2026

Own FP4-CSF loading and preserve native DS4.1 expert precision local-inference-lab/vllm#956

Closed

voipmonitor changed the title ~~Add NVFP4-CSF decoding and compatible compressed-scale APIs~~ Decode canonical FP4-CSF scales and accelerate NVFP4 TP2 serving Oct 1, 2026

This was referenced Oct 1, 2026

[KK] Kimi runtime, FP4-CSF and GLM online Spark precision: merge inventory local-inference-lab/vllm#906

Open

[beta] Integrate CSF preparation, native expert precision and decoder optimizations #456

Merged

voipmonitor and others added 2 commits October 1, 2026 16:06

voipmonitor changed the title ~~Decode canonical FP4-CSF scales and accelerate NVFP4 TP2 serving~~ Prepare and decode MXFP4-CSF/NVFP4-CSF expert scales Oct 1, 2026

Preserve native MXFP8 expert execution for compressed MXFP4 scales

7dc9174

voipmonitor changed the title ~~Prepare and decode MXFP4-CSF/NVFP4-CSF expert scales~~ Prepare canonical FP4-CSF scales and preserve native expert precision Oct 1, 2026

voipmonitor added 2 commits October 1, 2026 23:23

voipmonitor changed the title ~~Prepare canonical FP4-CSF scales and preserve native expert precision~~ Prepare FP4-CSF scales and repair expert batch decoding Oct 1, 2026

This was referenced Oct 4, 2026

W4A8: read MXFP4-CSF expert scales inline in the compact kernels (canonical twin of #479) #481

Merged

Karmic beta → canonical: PRs to merge local-inference-lab/vllm#984

Open

lukealonso added 2 commits October 5, 2026 22:59

Merge master into CSF scale preparation

4bb63bf

fix(quantization): validate paired CSF scale storage

74e6b01

lukealonso changed the base branch from feat/ds41-x4t-native-scale-decode to master October 6, 2026 06:03

lukealonso merged commit d7937d7 into master Oct 6, 2026

Navigation Menu

Repository navigation

Prepare FP4-CSF scales and repair expert batch decoding - #450

Prepare FP4-CSF scales and repair expert batch decoding#450
lukealonso merged 17 commits into
masterfrom
feat/nvfp4-csf

voipmonitor commented Oct 1, 2026 •

edited by lukealonso

Loading

Uh oh!

coderabbitai Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

voipmonitor commented Oct 1, 2026 • edited by lukealonso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

voipmonitor commented Oct 1, 2026 •

edited by lukealonso

Loading

coderabbitai Bot commented Oct 1, 2026 •

edited

Loading