Reduce NVFP4-CSF scale reconstruction overhead by voipmonitor · Pull Request #459 · local-inference-lab/b12x

voipmonitor · 2026-10-02T01:40:05Z

Reconstruct NVFP4-CSF scale operands for the rows consumed by each MMA warp group. Pair gate/up reconstruction barriers, mask partial consumer tiles, and use four-word indexed reconstruction in the standalone decoder. The inline program cache key is versioned to preserve kernel identity.

Validation on RTX PRO 6000 Blackwell Max-Q: 141 composed decoder/MoE tests pass; two cross-device tests skip in the single-device run. Sixteen indexed-vector cases pass memcheck with zero errors. All four inline consumer tile sizes (M16/32/64/128) pass unfiltered racecheck with zero hazards, errors or warnings, using --racecheck-report hazard --show-backtrace no. Earlier default-reporting attempts stalled inside the sanitizer host library during Torch reference setup and were interrupted.

Decoder-only balanced cold-L2 timing, 15 rounds of 16 samples, lower microseconds is better. Baseline was repeated after the candidate; every arm matches the exact byte oracle.

Geometry	Routed experts	Baseline us	Candidate us
E512/H2560/I320	10	6.723	4.825
E512/H2560/I320	40	9.172	7.109
E512/H2560/I320	63	12.012	8.931
E288/H4096/I512	10	6.801	6.190
E288/H4096/I512	40	16.532	11.733
E288/H4096/I512	63	23.416	17.278

These are synthetic decoder measurements. Full-model serving was not rerun locally; the historical serving artifacts retain their source identities.

Retain exact FP4 nibbles and caller-owned scale buffers. Decode only routed experts and pair gate/up and down scales without adding streams or PDL. Preserve native weight planning and existing Kimi X4T behavior. Qualify the master port with 67 tests and real-layer graph replay. Include serving-composition timing evidence and explicit qualification limits.

Slice native E2M1 experts and compressed UE8M0 scales without requantization. Allow Kimi TP1/2/4/8/12/16 geometry and reuse the paired scale decoder. Qualified serving geometry is TP16; other ranks require separate model qualification. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Retain all paired-scale and TP12 route ABIs through startup planning and register their caches for deferred-program reclamation. Native weight declarations and kernel arithmetic are unchanged. Validation: 75 X4T tests; real Kimi TP16 shard with autotuning and eight poisoned graph replays at each planned size from 1 to 1024 tokens.

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Co-authored-by: OpenAI Codex <noreply@openai.com>

…cratch Preserve native NVFP4 arithmetic and load TP extents without expanding resident scale storage. Retain paired CuTe programs for both routing integer types and qualify exact bytes and poisoned graph replay. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

Preserve X4T and LSC aliases, checkpoint bytes and kernel implementations. Document one-bit MXFP4 and four-bit NVFP4 offsets from a row base byte. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <vit@voipmonitor.org>

Require CSF schema and tensor names, preserve TP slices and FP32 calibration, and support Qwen3.8-Flash-Next geometry. Reorder resident offsets for contiguous decoding and fuse micro-kernel barrier initialization. GPU component validation: 79 tests passed.

Consume lazy projection sources and explicit geometry in B12X. Keep model names, manifests and file ownership in the companion vLLM loader. Preserve checkpoint bytes, scale layout, FP32 calibration and caller-owned scratch. Qualify with 88 component tests and byte parity for complete layers from Kimi, DeepSeek, GLM and Qwen. Co-authored-by: Codex <noreply@openai.com>

Accept rank-local CPU scale planes in prepare_weights and create resident scale batches inside its existing MXFP4/NVFP4 branches. Move checkpoint I/O and TP slicing to the paired vLLM integration. Preserve decoder kernels, calibration, scratch ownership and checkpoint bytes. Validation: 69 B12X tests; 56 vLLM tests; ten complete layer/rank comparisons with 132 byte-identical prepared tensors. Full-model serving was not rerun. Co-authored-by: Codex <noreply@openai.com>

Select cooperative shared-slot reconstruction or indexed plane expansion during typed preparation. Preserve native scale bytes, A4 arithmetic, N64 tails, caller-owned scratch and frozen replay. Correct the native route-broadcast lifetime and deterministic shared-activation indexing. Declare scale and reduction programs before preparation freezes kernel resolution. Validation: 408 component and native-oracle tests, two strict preparation declaration tests, and nine registry checks pass. Serving measurements and their performance limits are recorded separately.

Record matched GLM TP4 and Qwen TP2 MTP measurements, source and telemetry hashes, prepared memory use, and component correctness. Separate output throughput from speculative step-rate diagnostics and retain C1/Qwen residual costs. Describe the numeric scale-operand layout and independent native MoE invariant fixes. Timing remains research-only under the recorded NVML 0x400 hardware condition.

Pair ready operands before consumer synchronization, decode only consumed gate halves, and select partial-atom zero padding inside the shared reader. Preserve native MMA loads, checkpoint bytes and prepared memory sizes. Qualify 105 byte/expert graph tests and four online-preparation composition cases.

coderabbitai · 2026-10-02T01:40:13Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: local-inference-lab/b12x/.coderabbit.yaml
Review profile: CHILL
Plan: Advanced
Run ID: 5e53c0b1-7ae5-4d13-94c9-3f754ff9cd35

📥 Commits

Reviewing files that changed from the base of the PR and between 2877e75 and 567b3b8.

📒 Files selected for processing (73)

.lil/changes/b12x-450-csf-canonical.json
.lil/changes/b12x-450-csf-mxfp8.json
.lil/changes/b12x-450-csf-preparation.json
.lil/changes/b12x-450-csf-tensor-sources.json
.lil/changes/b12x-450-moe-routing.json
.lil/changes/b12x-450-nvfp4-csf-cooperative.json
.lil/changes/b12x-csf-indexed-vectors.json
.lil/changes/b12x-csf-operand-reconstruction.json
.lil/changes/b12x-kimi-x4t-prepared-scales.json
b12x/_lib/quant/csf_routing.py
b12x/_lib/quant/mxfp4_csf.py
b12x/_lib/quant/nvfp4_csf.py
b12x/_lib/quant/nvfp4_csf_inline.py
b12x/_lib/quant/x4t_packed_scales.py
b12x/_lib/quant/x4t_scales.py
b12x/moe/_shared/execution.py
b12x/moe/_shared/kernels/dynamic.py
b12x/moe/_shared/kernels/silu.py
b12x/moe/_shared/kernels/w4a16/kernel.py
b12x/moe/_shared/kernels/w4a16/prepare.py
b12x/moe/fused_moe/__init__.py
b12x/moe/fused_moe/_impl.py
b12x/moe/fused_moe/_preparation.py
b12x/moe/fused_moe/_tuning.py
b12x/moe/fused_moe/api.py
b12x/moe/fused_moe/planning.py
b12x/moe/fused_moe/weights.py
docs/lossless-scale-compression.md
tests/moe/test_mxfp4_csf.py
tests/moe/test_nvfp4_csf.py
tests/moe/test_nvfp4_csf_dispatch.py
tests/moe/test_w4a8_migration_corpus.py
tests/preparation/test_tuning_predicates.py
tests/preparation/test_x4t_moe_programs.py
tests/quantization/test_csf_routing.py
tests/quantization/test_mxfp4_csf.py
tests/quantization/test_nvfp4_csf.py
tests/quantization/test_nvfp4_csf_inline.py
tests/quantization/test_x4t_packed_scales.py
validation/csf/README.md
validation/csf/compare_tensor_loaders.py
validation/csf/dynamic-moe-correctness.md
validation/csf/mxfp8-serving.md
validation/csf/nvfp4-scale-operands.md
validation/csf/nvfp4-serving-evidence.json
validation/csf/nvfp4-serving.md
validation/csf/preparation-evidence/b12x-api-baseline-lint.json
validation/csf/preparation-evidence/b12x-api-lint.json
validation/csf/preparation-evidence/b12x-prepare-tests.txt
validation/csf/preparation-evidence/ds-prepared-parity.json
validation/csf/preparation-evidence/glm-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp12-prepared-parity.json
validation/csf/preparation-evidence/kimi-tp16-prepared-parity.json
validation/csf/preparation-evidence/qualification.json
validation/csf/preparation-evidence/qwen-prepared-parity.json
validation/csf/preparation-evidence/vllm-prepare-hooks.txt
validation/csf/preparation-evidence/vllm-prepare-mypy.txt
validation/csf/preparation-evidence/vllm-prepare-tests.txt
validation/csf/qwen-serving-evidence.json
validation/csf/qwen-serving.md
validation/csf/tensor-source-evidence/README.md
validation/csf/tensor-source-evidence/b12x-tests.txt
validation/csf/tensor-source-evidence/capture_layer_parity.py.txt
validation/csf/tensor-source-evidence/ds-layer-parity.json
validation/csf/tensor-source-evidence/glm-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp12-layer-parity.json
validation/csf/tensor-source-evidence/kimi-tp16-layer-parity.json
validation/csf/tensor-source-evidence/portable-tool-qwen.json
validation/csf/tensor-source-evidence/qualification.json
validation/csf/tensor-source-evidence/qwen-layer-parity.json
validation/csf/tensor-source-evidence/vllm-mypy.txt
validation/x4t/serving.json
validation/x4t/validate_layer.py

 _________________________________
< WMD: Weapons of Mass Debugging. >
 ---------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Record exact source and artifact identities, 105 correctness cases, racecheck and online-preparation composition. Publish generated-token and engine-step deltas separately, retain the TP2 C1 residual, and scope all timings to the observed clock-event mask.

…ression Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

…461) The publisher rejects an empty models list, so this fragment as published in the beta (fixed there by #461) is the one a canonical merge must carry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…006T052218Z-2bca98-459

voipmonitor and others added 15 commits September 29, 2026 22:54

Merge remote-tracking branch 'canonical/master' into review/kimi-x4t-…

c62107a

…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py

Declare Kimi compressed-scale preparation for runtime releases

552da21

Co-authored-by: OpenAI Codex <noreply@openai.com>

Preserve FP32 NVFP4 calibration under BF16 model loading

8cb43b9

Preserve native MXFP8 expert execution for compressed MXFP4 scales

7dc9174

voipmonitor marked this pull request as ready for review October 2, 2026 01:51

voipmonitor mentioned this pull request Oct 2, 2026

[KK] Kimi runtime, FP4-CSF and GLM online Spark precision: merge inventory local-inference-lab/vllm#906

Open

voipmonitor added 2 commits October 2, 2026 02:18

Vectorize indexed NVFP4 scale reconstruction and exception lookup

e646d0d

Qualify indexed scale reconstruction with Qwen TP1 and TP2 serving

566e288

This was referenced Oct 2, 2026

[beta] Integrate CSF preparation, native expert precision and decoder optimizations #456

Merged

fix(moe): resolve output determinism before candidate selection #465

Merged

docs: qualify CSF operand reconstruction without load-time scale comp…

993eac6

…ression Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>

voipmonitor mentioned this pull request Oct 4, 2026

W4A16: read NVFP4-CSF expert scales per pipeline stage (canonical twin of #471) #475

Merged

voipmonitor mentioned this pull request Oct 5, 2026

Karmic beta → canonical: PRs to merge local-inference-lab/vllm#984

Open

lukealonso added 3 commits October 5, 2026 23:04

Merge master into NVFP4-CSF reconstruction

5bd0dc2

Merge remote-tracking branch 'origin/master' into review/frozen-20261…

d05502e

…006T052218Z-2bca98-459

Merge remote-tracking branch 'origin/master' into review/frozen-20261…

826eb42

…006T052218Z-2bca98-459

lukealonso changed the base branch from feat/nvfp4-csf to master October 6, 2026 07:02

lukealonso merged commit 982fbbe into master Oct 6, 2026

Navigation Menu

Repository navigation

Reduce NVFP4-CSF scale reconstruction overhead - #459

Reduce NVFP4-CSF scale reconstruction overhead#459
lukealonso merged 23 commits into
masterfrom
perf/nvfp4-csf-operands

voipmonitor commented Oct 2, 2026 •

edited by lukealonso

Loading

Uh oh!

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

voipmonitor commented Oct 2, 2026 • edited by lukealonso Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

voipmonitor commented Oct 2, 2026 •

edited by lukealonso

Loading

coderabbitai Bot commented Oct 2, 2026 •

edited

Loading