This repository was archived by the owner on Oct 6, 2026. It is now read-only.
Repository navigation
[beta] Integrate CSF preparation, native expert precision and decoder optimizations - #456
Merged
Merged
Conversation
The preparation-session path (77351c1) resolves RoCE declarations through _dtypes(), which normalizes caller strings like 'float16' to torch.dtype objects, then compile_roce passes them straight into get_launcher. get_launcher's signature is dtype_name: str and _RoceOneshotLaunch rejects anything else against _DTYPE_PACK_ELEMS, so every weights-stage RoCE preparation dies with "unsupported RoCE one-shot dtype torch.float16" before the first collective. The old runtime path converted objects to names via _DTYPE_NAMES in roce_oneshot._launcher_key; the preparation path bypasses it. Normalize torch.dtype objects to their names at get_launcher entry, matching the conversion the runtime path already applies. Both call forms now reach the kernel gate; string behavior is unchanged. Reproduced and verified host-side: torch.float16 object clears the dtype gate post-fix (raising only the orthogonal threads check), and the string form is byte-identical in behavior. Pre-existing E501s and stale PCIe comm tests are untouched by this change.
…alues
Eight PCIe compile surfaces return the plain launch closure over their
annotated native program without attaching the program identity:
_dcp_a2a_cute: lse_reduce_scatter, all_gather_heads, all_gather_pair,
kimi_topk16
_dcp_topk_cute: owner stage
_hierarchical_cute: hierarchical launcher
_island_rs_cute: island reduce-scatter launcher
_vocab_argmax_cute: vocab argmax launcher
describe_compilation and every evict sweep call program_keys() over
those values; a plain function matches nothing, so any unit whose
CompileJob resolves one of these surfaces, or any evict_unretained pass
that reaches a cache holding one, dies with 'compile factory returned
an unannotated function'. On GB10 this surfaced mid norm.mhc racing
because the compile pool and eviction sweep span all registered caches,
not just the racing unit.
Wrap each closure with attach_programs(run, raw) so it carries its
program into retention and sweep keys. Additionally, program_keys now
treats plain functions as opaque retained values instead of raising:
a launcher closure produced outside a compile factory has no program
identity to lose, and eviction must never kill a live race. Factories
that return a carrierless closure still fail describe_compilation's
empty-carry guard, so the preparation contract is unchanged.
Verified host-side: plain function sweeps to (), wrapped closure
carries its ProgramKey, dict-of-annotated factories unchanged,
describe_compilation still rejects carrierless factories, exotic
unannotated objects still raise. Preparation host lane: 490 passed,
1 failed (test_device_reclaim selection-source assert) which fails
identically without this change.
compile_twoshot_surface resolves FP8 surfaces through get_twoshot_launcher, which returned a bare closure, so describe_compilation saw zero program carriers — same class as the RoCE and PCIe one-shot gaps. The BF16 getters already attach.
A collective authorization is consumed in the round after the exchange, and the local work before either rank launches (autotune races) is unbounded, so the kernel-side spin window ended up covering round time instead of launch jitter. Sessions now accept an optional collective_barrier(key, ranks) hook; when a blocked CollectiveRequirement is authorized, the job waits for every participant rank to enter before resuming. Timeout budget defaults to 120s via B12X_COLLECTIVE_BARRIER_TIMEOUT; timeouts raise CollectiveBarrierTimeout carrying key, ranks, and arrived sets.
torch's cpp_extension staleness check is mtime-based and does not survive a persisted TORCH_EXTENSIONS_DIR, so a changed _memory.cpp next to an old cached .so loaded stale symbols (missing release_graph_pool_cache). Key the extension name by the cpp content hash so any source change gets a fresh build directory.
get_launcher accepts str or torch.dtype. The torch.dtype spelling went through a bare dict lookup, so an unsupported dtype raised KeyError while the string spelling raised the documented ValueError from _RoceOneshotLaunch. Convert the lookup failure to the same ValueError.
Retain cross-rank collective priming and compiler program carriers while preserving master-scoped reclamation and RoCE shared-result ownership. Selected preparation, layout and RoCE tests pass (209 cases in the Kimi composition).
Retain exact FP4 nibbles and caller-owned scale buffers. Decode only routed experts and pair gate/up and down scales without adding streams or PDL. Preserve native weight planning and existing Kimi X4T behavior. Qualify the master port with 67 tests and real-layer graph replay. Include serving-composition timing evidence and explicit qualification limits.
Slice native E2M1 experts and compressed UE8M0 scales without requantization. Allow Kimi TP1/2/4/8/12/16 geometry and reuse the paired scale decoder. Qualified serving geometry is TP16; other ranks require separate model qualification. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Retain all paired-scale and TP12 route ABIs through startup planning and register their caches for deferred-program reclamation. Native weight declarations and kernel arithmetic are unchanged. Validation: 75 X4T tests; real Kimi TP16 shard with autotuning and eight poisoned graph replays at each planned size from 1 to 1024 tokens.
…backend-publication # Conflicts: # b12x/moe/fused_moe/api.py
…aration-publication # Conflicts: # b12x/preparation/session.py
Co-authored-by: OpenAI Codex <noreply@openai.com>
…cratch Preserve native NVFP4 arithmetic and load TP extents without expanding resident scale storage. Retain paired CuTe programs for both routing integer types and qualify exact bytes and poisoned graph replay. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Preserve X4T and LSC aliases, checkpoint bytes and kernel implementations. Document one-bit MXFP4 and four-bit NVFP4 offsets from a row base byte. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Martin Vit <vit@voipmonitor.org>
Require CSF schema and tensor names, preserve TP slices and FP32 calibration, and support Qwen3.8-Flash-Next geometry. Reorder resident offsets for contiguous decoding and fuse micro-kernel barrier initialization. GPU component validation: 79 tests passed.
Carry partial_dtype through dense MLA declarations, tuning queries, scratch and kernels. FP32 avoids intermediate BF16 rounding and doubles partial-output scratch. Defaults and split selection remain unchanged. Validation: Four host dtype contract cases pass. Five dense MLA implementation ASTs match the qualified e3e4fff4cf source. No GPU test was run in this audit environment. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com>
* Prepare EXL3 A16 experts and bounded W4A16 prefill Preserve source-coordinate expert extents, retained two-bit lookup tables and declared row capacities. Keep block-codec epilogues and pipeline tuning intact. Optional FP32 atomic prefill reduction does not guarantee bitwise summation order. Validation: Host eligibility checks pass for packed, ModelOpt, EXL3 and three block codecs. Syntax and whitespace checks pass. GPU validation of the canonical-base composition remains pending. Co-authored-by: Martin Vit <martin@voipmonitor.org> Co-authored-by: Codex <noreply@openai.com> * Share trellis preparation across native and EXL3 containers Normalize uniform EXL3 slots into TrellisSource and TrellisWeights. Preserve codewords, FP16 scale boundaries, global Hadamard coordinates and bounded CUDA staging. Route the compatibility loader through common preparation while retaining asymmetric-pair support. Native config callers keep their schema and projection-tiered path. CPU layout and adapter validation: 98 tests passed. GPU preparation, graph replay, performance and full-model fidelity remain unqualified in this environment. Assisted-by: OpenAI Codex * Keep common trellis preparation on compact decoding and atomic route packing Remove the full-codeword shared LUT and stable per-expert route ordering from the preparation interface change. Retain exact route capacities, prepared launch ownership and opt-in FP32 prefill reduction. Bump the MoE query schema for the reduced control set and scope the release fragment to PR 433. Validation: selected host suites report 181 passed and 28 failures; the same 28 missing-decode_config failures reproduce at 7fca2aa. Ten LUT-specific cases move with the LUT change. Changed Python sources parse and git diff --check passes. GPU execution and serving qualification were not run. Assisted-by: OpenAI Codex * Isolate the prepared W4A16 full-codeword LUT experiment Restore the 64 KiB shared codeword table, immutable selection control, residency checks and decoding tests as an independently reviewable change. Preserve the original auto/compact/full behavior; qualification remains pending. Document the instruction-scheduling difference that a matched compact-decoder benchmark must control. Validation: adapter, config, schema and variant-selection suites: 170 passed. Python source matches the extracted implementation except for the query schema version; syntax and whitespace checks pass. CUDA execution, graph replay and performance were not run. Source-commit: 7fca2aa Assisted-by: OpenAI Codex --------- Co-authored-by: Martin Vit <martin@voipmonitor.org> Co-authored-by: Codex <noreply@openai.com>
…Kimi MXFP4-CSF, GLM/Qwen NVFP4-CSF)
Master's 1.5.0 release bump arrives with the master sync. vLLM's b12x extra pins b12x==1.3.0 on both the beta and the canonical branch, so beta keeps the version it publishes until vLLM moves the pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Select cooperative shared-slot reconstruction or indexed plane expansion during typed preparation. Preserve native scale bytes, A4 arithmetic, N64 tails, caller-owned scratch and frozen replay. Correct the native route-broadcast lifetime and deterministic shared-activation indexing. Declare scale and reduction programs before preparation freezes kernel resolution. Validation: 408 component and native-oracle tests, two strict preparation declaration tests, and nine registry checks pass. Serving measurements and their performance limits are recorded separately.
Record matched GLM TP4 and Qwen TP2 MTP measurements, source and telemetry hashes, prepared memory use, and component correctness. Separate output throughput from speculative step-rate diagnostics and retain C1/Qwen residual costs. Describe the numeric scale-operand layout and independent native MoE invariant fixes. Timing remains research-only under the recorded NVML 0x400 hardware condition.
Add opt-in lossless GPU packing with caller-owned reconstruction scratch. Preserve activation arithmetic and transfer source ownership after preparation. Qualify codec bytes, native expert graphs, and native-checkpoint serving. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Signed-off-by: Martin Vit <martin@voipmonitor.org>
Pair ready operands before consumer synchronization, decode only consumed gate halves, and select partial-atom zero padding inside the shared reader. Preserve native MMA loads, checkpoint bytes and prepared memory sizes. Qualify 105 byte/expert graph tests and four online-preparation composition cases.
Record exact source and artifact identities, 105 correctness cases, racecheck and online-preparation composition. Publish generated-token and engine-step deltas separately, retain the TP2 C1 residual, and scope all timings to the observed clock-event mask.
Preserve prepared bytes and maximum-coverage palette ties while removing CPU exception indexing and amortizing short-row packing across a CTA. Checkpoint codecs and execution kernels are unchanged. Validation: 166 GPU component tests, 24 memcheck cases, exact CPU-index and native-byte oracles, and balanced real Kimi TP16 scale measurements. Signed-off-by: Martin Vit <martin@voipmonitor.org>
Record exact byte oracles, GPU index checks, complete native startup probes, and balanced component timings with explicit measurement limits. Provide a real-scale probe for Kimi TP16 and GLM TP2; checkpoint formats and execution precision remain unchanged. Signed-off-by: Martin Vit <martin@voipmonitor.org>
Qualify real layer scale reconstruction and prepared-buffer parity against the CPU index. Record a 36.06x Qwen TP1 preparation gain and unchanged DS4.1 TP4 timing, with whole-model startup scope kept separate. Signed-off-by: Martin Vit <martin@voipmonitor.org>
Signed-off-by: Martin Vit <martin@voipmonitor.org>
Retain native MXFP8 and Trellis eligibility rules with compressed-scale tuning under query schema 17 and configuration schema 10. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Signed-off-by: Martin Vit <martin@voipmonitor.org>
…ition Retain online preparation and dynamic tile coverage in the shared native-output oracle. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Block codecs retain their resource-valid tuning choices without an unused Trellis table. Preserve Trellis compact and complete codeword selection. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Record paired runtime identities, block-codec resource invariants, GPU validation and the separate public-image serving scope. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
voipmonitor
changed the title
voipmonitor
marked this pull request as ready for review
October 2, 2026 09:25
Contributor
Author
Serving evidence for public beta image
|
| Profile | Stored CSF C8 / C16 | Online CSF C8 / C16 |
|---|---|---|
| Qwen TP1 / MTP3 | +0.64% / +0.07% | -2.08% / +0.08% |
| Qwen TP2 / MTP3 | -1.85% / -1.67% | +0.23% / -3.04% |
| DS4.1 TP4 / DSpark K7 | -1.60% / -2.17% | -2.27% / -3.28% |
| GLM TP4 / MTP3 | -2.38% / +0.85% | -1.28% / +0.13% |
| GLM TP2 / no speculation | -1.96% / Unsupported capacity | -1.92% / Unsupported capacity |
| DS4F 0731 TP2 / no speculation | Not tested | -2.66% / -2.70% |
| Kimi TP16/DCP16 / no speculation | -1.78% / -2.41% | -1.81% / -2.10% |
Remaining limits: GLM TP2 without speculation retains C1 losses of 5.86% (stored) and 5.93% (online). DS4.1 and Qwen have larger speculative C1 output losses than their verification-rate changes; the report includes both metrics and does not infer quality from accepted length. GLM TP2 C16 cannot sustain 16 resident requests in the 3 GiB cache pool. Native Kimi with 3 GiB KV/rank failed warmup, so matched tests use 1 GiB. Failed/partial attempts remain in the archive. Full-model KLD, equal greedy streams and 1M-context behavior are unmeasured.
Batch output changes in the qualified C8/C16 cells range from −3.28% to +0.85%; this is bounded evidence for these workloads, not universal throughput parity. No canonical feature PR, issue #906 or launcher #119 was modified by this validation.
Complete report · Measurements and comparisons · Raw receipts, harness sources and SHA256 manifest
Archive SHA256: 1060dff48fc283ff63b6a0d5505139661b7d3f250007afca9f6e6c4d6c477107.
voipmonitor
added a commit
that referenced
this pull request
Oct 3, 2026
[beta] Integrate CSF preparation, native expert precision and decoder optimizations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
integration/karmic-kraken-betaruntime prepares canonical MXFP4-CSF/NVFP4-CSF scales through ordinary fused-MoE weight processing. vLLM owns checkpoint discovery and TP slicing. DS4.1 retains native MXFP8 expert activations; Kimi retains BF16 expert activations. Native checkpoints support opt-in lossless scale preparation, and NVFP4 decoding shares reconstruction work for larger batches and short routes.This integrates #450, #458 and #459 into the existing beta port, paired with vLLM #963. It retains #384 preparation ownership, the master attention/Trellis changes (#441/#436/#442), native beta W8A8 support and the beta B12X 1.3.0 package contract. The canonical implementation PRs remain separate review targets.
Implemented: numeric CSF preparation, native MXFP8-CSF execution, GPU online encoding/index construction and optimized NVFP4 reconstruction. Native and stored checkpoint weight values, scale codecs and expert arithmetic are preserved. Shared reconstruction scratch requires serialized layer execution. The vLLM online flag defaults to off.
MoE tuning combines the native MXFP8 and Trellis eligibility rules with CSF controls under query schema 17/configuration schema 10. Only Trellis weights reserve a Trellis lookup table: IQ2_XS and Q8_0 candidates retain their validated shared-memory capacity without reducing candidate coverage. Each behavior has an immutable release fragment.
Qualified component validation: 466 cases in the combined suite passed; both resource failures reproduce on frozen parent
8f0f0182and are corrected by the table-ownership guard. The affected block-codec/Trellis/CSF/MLA suite passes 139 cases. Paired vLLM passes 61 loader cases and 29 staging/online/native-MXFP8 adapter cases; its applicable pre-commit hooks including mypy pass. Release-fragment dependencies resolve and published fragments are unchanged.Commands, exact source/image identities, the parent reproducer and receipt hashes are in the component qualification report. These component gates use CUDA 13.4/CUTLASS DSL 4.7.1 on RTX PRO 6000 Blackwell. Public CI-built Docker serving and speed measurements remain pending and will be recorded on this integration PR after publication; source-overlay results are not presented as that qualification. Existing preparation/serving measurements retain their separately pinned, research-only scope. Full-model KLD and maximum-context stress remain unmeasured.
AI assistance: integration, conflict resolution and validation used OpenAI Codex at the repository owner's request. Human maintainer review is not represented as complete.