Conversation
Keep fixed-size shared route staging on fused W4A16 launches, while the FC2-only endpoint validates every runtime-M route against the caller's resident expert count. Invalid routes address expert zero with an exact-zero effective weight without mutating caller tensors or allocating during CUDA Graph replay.\n\nPass the resident expert count separately from the compile-time expert-capacity bucket so matrix-granular endpoints retain kernel-cache reuse. Cover M=3 narrow and wide FC2, M=7 FC2, invalid negative and upper-bound IDs, eager execution, CUDA Graph replay, and immutable route inputs.
Add an opt-in large-token-count W4A16 FC2 epilogue that reduces routed BF16 outputs directly into a caller-owned FP32 token accumulator and casts the assembled result once. The arena planner replaces the token-by-route-by-hidden output extent with token-by-hidden BF16 output storage plus token-by-hidden FP32 accumulation storage. Small-token-count tensor-core decode, FP16/full-rotation execution, and activation-amax capture retain their existing launch and workspace contracts. The prefill path uses relaxed FP32 global reductions and is disabled unless B12X_W4A16_PREFILL_FUSED_SUM is set. Validated with 221 passing W4A16 GPU tests, fixed-scratch CUDA Graph replay, NaN-poisoned accumulator initialization, Kimi-K3 TP16 shape numerics, and full-model 4,096-token scheduler chunks.
…e W4A16 microkernels
실행 컨테이너의 B12X 소스와 기본 revision의 차이를 보존한다. 원본 파일의 경로와 SHA-256은 tp9-colocated-qsrt-20260905/evidence/implementation-source-base.json에 기록되어 있다. GPU 성능 검증을 주장하지 않는다.
Preserve all peer pointers in BF16 two-shot collectives, use a rank-invariant accumulation order for TP9, and accept head-tail capacity in DCP buffers. Assign complete QSRT H128 extents without padded expert storage and rotate rank ownership across layers. Validation: 314 focused tests passed. Real QSRT extents at widths 256 and 384 produced bit-identical outputs for decode and 1536-token prefill. Physical nine-GPU serving performance is not yet qualified. Co-authored-by: Codex <codex@openai.com>
Keep Kimi-K3 hidden widths on the BF16 two-shot and DMA paths by packing only the collective wire tail. Preserve out-of-place output lifetime and include padding in cached DMA graphs. Other supported world sizes retain their divisibility checks. Validation: nine physical RTX PRO 6000 GPUs passed oneshot, two-shot, DMA and DCP eager/graph checks, including 1536 x 7168 prefill, ninth-rank mutations, head-tail views and an empty KV shard. Full-model TP9 performance remains unqualified. Co-authored-by: Codex <codex@openai.com>
The pull all-reduce kernel takes a remainder pack count and assigns each rank a contiguous shard whose length differs by at most one pack, so a payload whose 16-byte pack count is not a multiple of the world size is reduced in place. Single-pack rows (row_elems=8) use this partition; wider rows keep the row-aligned equal shards and the wire-padded path. Scope: nine-rank Kimi-K3 decode payloads (T x 7168 and T x 3584 bf16) previously took the padded path for every T not divisible by nine, which added a copy-in, a zero fill and a copy-out launch per call and two wire buffers per runtime. Reduction order and fp32 single-rounding math are unchanged; outputs are bit-identical to the padded path. Validation: Python syntax checked. tests/comm/test_pcie_tp9_physical.py now reduces position-dependent payloads for 4/8/9/16/32 x 7168 and 4/16 x 3584 rows eagerly and under graph replay; that GPU run and benchmarks/benchmark_pcie_tp9_allreduce.py (one-shot vs two-shot vs NCCL per-call latency) are pending nine idle GPUs. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
`--m-values` lists the token counts to run and `--rank` picks the TP9 rank whose extent is loaded, so the compact-versus-padded comparison can target the serving batch shapes of a specific rank (for example 244 tokens on a rank that owns a 256-channel extent) instead of the fixed 1/2/4/8/16/1536 set on the first 256-channel rank. Defaults are unchanged. Validation: syntax checked; runs need a GPU and the checkpoint. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
benchmarks/benchmark_pcie_tp9_allreduce.py times the B12X one-shot pool, the BF16 two-shot runtime and PyNCCL for Kimi-K3 decode payloads (T rows of 7168 or 3584 bf16 elements) on the full nine-rank group, eagerly and under CUDA graph replay, after checking each path bit-exactly against the fp32 reference. Rank 0 prints one JSON record per path, shape and mode with median and p90 microseconds; a shape a path rejects is reported and skipped. Validation: syntax checked; needs nine idle GPUs. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
`PCIeTwoShotBF16.all_reduce_mode = "push"` selects a kernel whose remote traffic is posted PCIe writes only: each rank pushes its contribution to every peer's shard into the peer's staged payload region, reduces its own shard from local memory after the first barrier, pushes the bf16 result into every peer's reduced region, and copies the peers' reduced shards from local memory after the second barrier. The staged slot's reduced region now holds one shard per source rank. The pull kernel (remote reads) stays the default; both use the balanced pack partition and the same fp32 reduction order, so their outputs are bit-identical. Scope: nine-rank Kimi-K3 decode all-reduces. The one-shot pulls the whole payload from all eight peers, which scales with bytes and peer count. Validation (nine RTX PRO 6000, PCIe, tests/comm/test_pcie_tp9_physical.py): pull and push modes bit-exact against the fp32 reference for 4/8/9/16/32 x 7168 and 4/16 x 3584 rows, eager and CUDA graph replay. benchmarks/benchmark_pcie_tp9_allreduce.py, graph replay, median us: 57 KB one-shot 66.8 / pull 38.3 / push 20.4; 114 KB 123.3 / 48.9 / 31.3; 229 KB 240.6 / 71.0 / 53.6; 28 KB 38.8 / 28.0 / 17.7; NCCL 46-105. The benchmark also creates one one-shot pool per shape because captured graphs need unique stream-affine channels. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
Nine RTX PRO 6000 form two PCIe switch clusters (ranks 0-3, 4-7) plus one GPU on the CPU root complex (rank 8). The BF16 two-shot serves this world with every rank pushing to all eight peers and two nine-way barriers, so each phase waits on the root link. PCIeIsland9AllReduce keeps traffic inside the clusters: rank island*4+lane owns quarter `lane`; ranks scatter their quarters to the island owners (rank 8 to the island-0 owners) with posted writes; owners reduce in fp32 in ascending source order, exchange fp32 partials with the same-lane owner of the other island, add them (island 0 first) and round once to bf16; owners push the final quarter to their island peers and, from island 0, to rank 8. Each rank maps at most five peers. Per-block generation flags double-buffer the inboxes, so the kernel replays in CUDA graphs without host involvement. The interface matches the two-shot runtime's (accepts, all_reduce(inp, out=), prepare_graph, capture, close) so the serving dispatcher can select it in place of the two-shot. Numerics: fp32 accumulation with one bf16 rounding, as the two-shot, in a different order ((0+1+2+3+8) + (4+5+6+7) instead of owner-first sequential). benchmarks/precision_pcie_tp9_allreduce.py measures both against the correctly rounded fp64 sum on heavy-tailed activations and reports the elements on which the kernels differ. Validation: tests/comm/test_pcie_tp9_physical.py gains the island9 section (position-dependent patterns for 1/2/4/8/9/16/32x7168, 4/16x3584, 1x1030; a total whose island-0 partial is not bf16-representable, 258 = 4x64 + 1 + 1; graph replay with a changing rank-8 input) and benchmark_pcie_tp9_allreduce.py times the kernel next to one-shot, two-shot pull/push and NCCL, now including 1 KB-14 KB payloads. Both need nine idle GPUs and run in the next maintenance window; not yet executed. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
…arness --time-iters N reports the median graph-replay time of each runtime width, --block-m accepts 96 and 128, and every record carries a SHA-256 prefix of the 384- and 256-width outputs so runs with different route blocks or decode knobs can be compared for bit-identical results without keeping tensors. Inputs are seeded, so the digests are comparable across processes. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
…s width The fused MoE planner (compile_w4a16_fused_moe) checks whether a candidate CTA tile fits shared memory with the B stage sized at weight_bits = max(4, trellis_bits). That width is the residency (blocks per SM) planning contract, not the staging layout: W4A16GemmKernel stages a native trellis3_t256 tile at 4 * trellis_bits bytes per 32 codes, so a 2-bpw tile stages half the bytes the fit check assumed. At the pinned 128x128 trellis tile the 64-row route block was therefore rejected at an estimated 103,424 B against the 100,864 B budget although the kernel's own layout needs 86,016 B (90,128 B with the 4 KiB T12 table and copy barrier), below the 101,376 B SM120 opt-in limit. _w4a16_b_unit_bytes now defines the staged bytes per 32 codes for every weight layout and pair kind; W4A16GemmKernel.b_unit_bytes and the planner's fit check both derive it from there. _shared_memory_footprint and _candidate_tile_fits accept that width (b_unit_bytes) and _select_tile_config passes it to the fit only (fit_b_unit_bytes), leaving _determine_blocks_per_sm on weight_bits, so tile choice, planned residency and the grid contract are unchanged for every geometry that fit before. With the real width the fit accepts route block 64 for the 2-bpw K3 layout (88,064 B estimate) and still rejects block 96 (130,048 B); the kernel and fused constructors keep their exact layout checks as the second gate. The serving side needs no change: kquant_hybrid tries blocks 64, 48, 32, 16 in order, so block 64 is selected once VLLM_KQUANT_W4A16_PREFILL_BLOCK_M is 64 or unset (select_route_block_size_m(4608, 16, 896) = 64). Route block is not part of the per-row accumulation order (tile_k, tile_n, cta_threads, warp K-slices), so block 64 is bit-identical to block 48 by construction. benchmarks/benchmark_qsrt_tp9_extent.py gains --topk-ids <file.pt> (replay a captured [tokens, 16] routing; token count from the file), --routing zipf:<s> (seeded over-dispersed synthetic routing) and a per-record routing histogram (rows per expert mean/p50/p90/max/empty, total route blocks, padding fraction and the fraction of experts needing 0/1/2/3/4+ blocks at the chosen block and at 48 and 64), next to the existing output_sha256 digests; the vLLM checkpoint reader import moves into main() so the routing helpers import without vLLM. Validation (CPU, production image, no GPU): tests/moe/test_w4a16_tile_selection.py (29 passed: unit widths for every layout/pair kind, 4-bpw vs real 2-bpw footprints at blocks 48/64/96 = 86,784/103,424/136,704 vs 70,400/88,064/130,048 B, fit accept/reject at the K3 FC1/FC2 shapes, unchanged _select_tile_config results, kernel layout 68,352/86,016 B at blocks 48/64, fused 90,128 B with the modal table, compile_w4a16_fused_moe admitting blocks 32/48/64 and rejecting 96 at the pinned tile) and tests/moe/test_benchmark_qsrt_tp9_extent_routing.py (4 passed). GPU digest and timing comparison of block 48 vs 64 is pending the next maintenance window. Co-Authored-By: Claude Code <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5 (cherry picked from commit 4232612)
benchmarks/benchmark_qsrt_tp9_extent.py --topk-ids accepts the capture
dict written from serving ({"topk_ids": int32 [tokens, 16], "layer": int,
"num_tokens": int}, optionally "topk_weights"): a num_tokens that differs
from the row count is rejected, the captured layer is recorded in the
record's routing source ("captured:<file>@layer<L>") and a mismatch with
--layer is printed, so a replayed histogram is attributed to the layer it
was taken from. Bare tensors and dicts without the metadata load as before.
Validation (CPU, production image, no GPU):
tests/moe/test_benchmark_qsrt_tp9_extent_routing.py and
tests/moe/test_w4a16_tile_selection.py, 33 passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011aPCVZBsteYgs4PTAQYuE5
(cherry picked from commit ac8572e)
…annel DCP a2a channels are stream-affine: preparing the eager channel on the capture stream raised on every rank. The eager paired checks already prepared the device-slot launcher variant, and `capture` prepares the new logical channel on the capture stream collectively, so the check needs no prepare call. Window 12 run 2: all nine ranks passed the paired eager checks under pull and raised on this call. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
Per-rank stage lines after every eager paired batch (full-width and clipped), before and after the paired graph capture and after each replay, so a rank that stops participating in the nine-rank collective sequence can be placed at the check it was in. Window 12 run 3 under push: eight ranks reached the paired graph check and rank 7 did not, with the NCCL watchdog reporting rank 7 eight to fourteen collectives behind the others. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
`B12X_PCIE_DCP_A2A_PAIR_TRANSPORT` (pull or push; unset inherits the runtime transport of `B12X_PCIE_DCP_A2A_TRANSPORT`) selects the transport of `all_gather_pair` and its fused Kimi top-k variant on their own, so the head gather and the LSE reduce-scatter can stay on push while the paired gather runs pull, or the reverse, for A/B measurement and staged deployment. The value enters the channel layout contract that every rank checks collectively, the pair launches, the graph prewarm and the capture checks. Validation: tests/comm/test_pcie_dcp_a2a.py (90 passed, CPU, serving image): the override parses, inherits and rejects unknown values, and the pair prewarm compiles the pull launcher while the head gather keeps push. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
A rank that raised inside a check ended in the pool teardown's barrier, which paired with the other ranks' next collectives, so every rank hung until the NCCL watchdog and the exception never reached the log (window 12 runs 3 and 4: one rank per run stopped advancing under push with its flight recorder showing a lone all-reduce). The worker now prints the exception as a stage line with the traceback and ends the process, so mp.spawn reports it at once. The paired projection gather checks move after the head/LSE eager, graph and mixed-grid checks so a failure in either family is attributable to it alone. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
The probe samples one staging word per slot around single gathers to verify that graph replays alternate slots. Under the push transport that word is written by a peer, and a peer that is ahead pushes its next launch's rows into the other slot before its barrier, so a rank sampling between launches could see both slots change (window 12 runs 3 to 5: rank 7, 4 and 0 failed the assertion; window 10b passed on timing). A device sync plus rank barrier before every sample and after every launch keeps all ranks between launches while sampling; the assertions are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
`--widths` (default 384,256) lists the runtime intermediate widths to build. A 384-channel extent (rank 0 at layer 1 under the nine-rank rotation) only fits the 384 width; the fixed (384, 256) sweep raised on its second width and produced no record for such ranks (window 12 MoE baseline step). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
…exist The per-m record compared the 256 width against the 384 width by name; a single-width run (a 384-channel extent with --widths 384) raised KeyError after its timings. The widest built width is now the reference and the narrowest is compared against it (identical when only one width exists), and the record lists the widths it built. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
…zation `B12X_MXFP8_LINEAR_FUSED_QUANT_A_SHAPES` lists `KxN` shapes (input features x output features) that take the fused small-M GEMM (activation quantized in-CTA) even when N exceeds `B12X_MXFP8_LINEAR_FUSED_QUANT_A_MAX_N`. The remaining conditions of the fused route are unchanged: at most eight rows, bf16 activations, unpadded K, and N a whole number of 64-wide tiles. The N cap approximates the trade-off between the fused GEMM (no split-K, the activation re-quantized per N tile) and the separate quantizer plus the split-K GEMM; it was set from a K = 7168 measurement. A narrow K streams little weight per CTA, so a shape such as the Kimi-K3 KDA output projection (K = 1536, N = 7168) can be cheaper on the fused route, which also removes one quantizer launch per projection. Both routes compute the same UE8M0 scales and E4M3 values, so the outputs are bit-identical; the list only changes which shapes skip the separate kernel. Unset, the rule is the previous one. Validation: tests/gemm/test_mxfp8_linear.py (the two rule tests: listed shapes route past the cap, unlisted and K-padded and fp16 and nine-row shapes do not, malformed entries raise; the empty default) pass on CPU. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
Contributor
Author
Maintenance window 12 (2026-09-07 15:12–16:27 UTC): C1 + C2 servedDecision: ship C1 (paired projection gather push transport) + C2 (register-held top-16 selection); the logical-width paired gather kernel ( Idle-GPU proofs (nine RTX PRO 6000)
Serving qualification (all passed): canaries, concurrency-1 samples, cold 8k/64k TTFT probes, cross-image vision isolation, LMCache arms Q1–Q7, prefix-hit/external/mixed prefill reproductions, prefill tail integrity, 220k cold prefill with a clean kernel log. Benchmark pair (
Decode steps/s +2.1…2.6 % at every context (token/s follows the draft acceptance of each 30 s sample and is not the metric); prefill within 0.5 % (the kernels are decode-only). Test-harness fixes on this branch found by the window: the nine-rank GPU test's slot-parity probe raced against peer pushes ( Served since 2026-09-07 16:11 UTC. |
The full-rotation top-k route sum (`W4A16TopKSumKernel`, one launch per MoE layer) accumulates each output element in fp32 and stored the fp32 value; a caller that wants the model dtype had to round it in a separate cast kernel. `B12X_W4A16_TOPK_SUM_OUTPUT` (`fp32`, the default; `bf16`; `fp16`) now selects the store element: the kernel rounds the same fp32 value once, to nearest even (`cvt.rn.bf16.f32` / `cvt.rn.f16.f32`), so the bytes equal the separate cast's and the cast launch disappears. The plain (non-rotation) sum keeps storing its element dtype. Plumbing: the kernel object carries `output_dtype` and casts at its three full-rotation store sites (`_cast_out`); `compile_w4a16_topk_sum` takes `output_dtype` (default from the setting under full rotation), keys the compile cache and the disk spec by it and records it on the compile result; the flat launch derives the variant from the output tensor's dtype and types its pointer accordingly; `run_w4a16_moe` checks the output tensor against the setting; the host scratch allocation follows the setting. Prewarm and eager launches read the same setting, so one process compiles one variant. Unset, the compiled code is the previous one. Validation: tests/moe/test_w4a16_topk_sum_output_setting.py (setting, kernel object, compile key, plain-sum rejection) and the rule tests of tests/gemm/test_mxfp8_linear.py plus tests/moe/test_w4a16_tile_selection.py, tests/gemm/test_launch_custom_ops.py and tests/moe/test_w4a16_route_pack_compile.py: 92 passed on CPU in the serving image. Offline: every full-rotation variant (plain and coupled Hadamard, per-expert and broadcast output scale, mapped and unmapped ids) compiles for the three store elements; the bf16 variants emit `cvt.rn.bf16.f32` with 16-bit stores; ptxas sm_120a assembles them (44 registers, no spills). tests/moe/test_w4a16_topk_sum_output_dtype_gpu.py (one GPU, B12X_RUN_W4A16_TOPK_SUM_OUTPUT_TEST=1) compares the bf16 and fp16 stores with the separate cast byte for byte across the variants. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
…ecks The fused MoE plan sized its full-rotation route-sum buffer (`full_rotation_output`) as fp32 and the bind, run and scatter entries required an fp32 output tensor, so a caller could not use the bf16 or fp16 store of the top-k sum kernel (`B12X_W4A16_TOPK_SUM_OUTPUT`). The buffer dtype and the three checks now come from the same setting (`_rotation_output_dtype`), and the error text names the expected dtype. Unset, the buffers and checks are the previous fp32 ones. Validation: tests/moe/test_tp_moe_scratch_bindings.py and tests/moe/test_w4a16_tile_selection.py pass on CPU in the serving image with the setting unset and with bf16 (the five failures of the bindings file are the branch base's stale `plan_b12x_fp4_moe_weights` keyword and fail identically on the previous commit). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
The mixed K3/K4 Trellis launch allocates fp32 route-sum buffers and types their pointers as fp32 when it calls the compiled top-k sum directly, so its compile must not follow the process-wide store-element setting (`B12X_W4A16_TOPK_SUM_OUTPUT`): with a bf16 setting the compiled kernel would expect a bf16 output pointer. Both compile sites now pass `output_dtype="fp32"` explicitly. No behaviour change with the default setting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
`B12X_W4A16_M8_CTA_THREADS=512` (default 256) doubles the warps of the M8 decode CTA: each warp owns two N16 blocks instead of four, so the tile's N-warps double while the K-slices, the per-warp MMA sequence and the fold order stay the same. Every output element is accumulated from the same fp32 operations in the same order, so the outputs are bit-identical to the 256-thread geometry; the scheduler has four warps per SM sub-partition instead of two to pick from while the shared-memory table lookups of the trellis decode are in flight (Nsight Compute of the served kernel at m = 4: 77 % of cycles with no eligible warp, 2 active warps per scheduler, shared-memory pipe 63 %). Kernel: `n_blocks_per_warp` (4 or 2) replaces the literal four N16 blocks per warp in the M8 tile path (trellis window loads, dequant/MMA bundle, split-K combine, K-slice fold, output staging, register bundles and accumulators); `warp_k16_iters` (K16 blocks per warp row and stage) is separated from `b_sh_wr_iters` (packed-B staging iterations), which only coincided at 256 threads; the fold scratch is sized from the warp-row thread count (`red_threads`, `red_stride`), unchanged at 256 threads. The geometry is defined for the M8 schedule over uniform-rate trellis3_t256 weights without paired routes; other configurations raise. The fused kernel and `compile_w4a16_fused_moe` carry `cta_threads_multiplier`, the compile entry reads the setting for `moe_block_size == 8`, and the register table gains the 512-thread entries. Validation: CPU plan and binding suites unchanged (125 passed, the five stale bindings failures); offline CuTe DSL compile of the served decode geometries (384-wide extent: 128 x 128 tiles; 256-wide: 256 x 64) at 256 and 512 threads, ptxas sm_120a: 117 / 96 and 115 / 92 registers, no spills, `.reqntid 512` on the doubled CTAs. Bit-identity against the 256-thread kernel (output and top-k digests of the extent benchmark at m = 1/2/4/8, both widths, uniform and Zipf routing) and the replay time are the idle-GPU proofs of the next maintenance window. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6pDrAEhR2rD6cEHyQRAS5
Contributor
Author
Maintenance window 13 (2026-09-07 16:40–17:08 UTC): G2 served, G4a closed, K-lever dataIdle-GPU proofs (one RTX PRO 6000)
Served since 2026-09-07 16:52 UTC: fingerprint Benchmark pair (before = the window-12 build measured 40 min earlier on the same host; valid matrix):
Cumulative since the campaign's served baseline ( |
`benchmark_qsrt_tp9_extent.py` allocated the full-rotation route-sum output as fp32; with `B12X_W4A16_TOPK_SUM_OUTPUT=bf16` in the environment (the serving setting) the fused MoE bind rejected it, so the decode measurements could not run under the served switches. The buffer now takes the kernel host's store element, as the serving runtime does, and the bind-path error names the expected dtype and the setting. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014od2j9nF5hV8bA2eos5YSB
Contributor
Author
Maintenance window 14 (2026-09-07 17:29–17:52 UTC): K1 proof not run; served set rebuilt on this branch's headThe window followed a host reboot (GPU 0 switched to compute mode with ECC off). Its idle-GPU steps failed immediately for two harness reasons, both fixed on this branch: the extent benchmark allocated its route-sum output as fp32 while the container environment carried Reduced qualification at the operator's request (canaries, prefill tail integrity, kernel log since boot — the other arms were not run). Benchmark vs the window-13 build: decode steps/s ×0.983…0.985 at every context (31.4 → 30.9 at ctx 0), prefill ×0.99–1.00. The shortfall is a clock policy artifact of the reboot, not the kernels: the static clock policy left GPU 4 locked at its 3,090 MHz table maximum (the other eight ranks ran unlocked with a 3,337 MHz ceiling); under load GPU 4 held 2,340 MHz against 2,520–2,587 MHz on the others and 2,580 MHz before the reboot, and TP9 paces at the slowest rank. The lock was reset ( |
Preserve individual replay samples and GPU operating state, optionally attest the four-row CUDA graph launch, and compute diagnostic error metrics in FP32. Kernel arithmetic is unchanged. Instrumented runs under concurrent serving retain correctness evidence but are excluded from performance decisions. Co-authored-by: OpenAI Codex <noreply@openai.com> (cherry picked from commit 3687304)
Keep raw launch and replay evidence for the Kimi-K3 TP9 source extents. Mark the thread-count comparison unqualified when the paired baseline overlaps serving. Add optional vLLM activity observations without changing the configured output dtype or kernel arithmetic. Validation: eight routing/activity CPU tests pass; all sixteen recorded GPU numerical cases are finite with matching cross-thread digests and exact eager/graph results. Ruff and diff checks pass. Instrumented timings are excluded from performance decisions. Co-authored-by: OpenAI Codex <noreply@openai.com>
myshytf
changed the title
Keep FP16 and BF16 fragment semantics while reducing the direct table byte-merge chain. Add opt-in per-CTA global-timer records with preplanned workspace, capacity checks, and pool-mode invalidation. Preserve offline object/resource auditing and correctness oracles. GPU qualification and end-to-end performance measurements remain pending.
Bind all eight width/thread/pair cases to source and artifact hashes. Baseline text matches four archived objects. Report register increases and keep GPU correctness and performance qualification pending.
Keep compact staging for both transports and use Int64 destination strides. Expose a layout capability for integrations and distinguish the launch ABI with compile specification 14. Add host layout checks and GPU replay, raw-byte and large-offset cases; GPU qualification remains pending.
myshytf
changed the title
…arness Record nine direct-LUT GPU checks and TP9 pull/push graph gates, including opaque packed records and large output strides. Add a research-only native-extent harness that shares weights between variants and records exact objects, mutations, hardware state and phase diagnostics. Native extent timings and full-serving performance remain pending.
Contributor
Author
|
K1 (
Nsight Compute (m = 4, 384-wide, profiler clock 2.17 GHz): 256 threads 138.1 µs, 512 threads 139.1 µs; executed instructions 48.9 M vs 56.9 M (+16 %); registers 121 vs 98, no spills; stall per issue-active: MIO throttle 3.44 vs 7.08, barrier 1.17 vs 2.66, long scoreboard 0.66 vs 0.75, short scoreboard 0.51 vs 0.89; L1/TEX pipe 62.5 % vs 64.9 %, DRAM 23 %. Reading: the decode launch is bound by the shared-memory instruction pipe of the direct-table lookups (MIO throttle), not by latency; doubling the warps lengthens that queue and duplicates per-warp A-fragment/metadata work. K1 is held (slower at both extents) and the pair decode is neutral (within noise) — neither enters the serving environment. The remaining exact lever for this launch is fewer table lookups per weight, not geometry. Evidence: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kimi-K3 TP9/DCP9 decode spends time gathering paired projections, selecting experts, and running separate quantization or output-cast kernels. This branch provides opt-in transport and kernel choices for those operations, plus numerical and measurement tooling. The branch uses the serving source lineage rooted at
71df6033; full integration into another lineage requires matching its collective and binding contracts.GitHub reports conflicts with
master. The published evidence applies to the serving-lineage source identified here; a merge-resolution port requires its own compatibility checks.Behavior
Query head gather accepts aligned, non-overlapping padded batch rows while retaining compact pull/push staging.
supports_all_gather_heads_output(out)lets integrations avoid an intermediate only for supported layouts; compile ABI 14 carries an Int64 output stride. Padding is preserved.B12X_SQG_XOR_CHEB_T12_DIRECT_PAIRS=1returns four independent byte pairs from the direct shared LUT. The native full-rotation path remains FP16 internally; model-facing BF16 input and the separate BF16 fragment conversion remain supported. The option is off by default.B12X_W4A16_PHASE_PROFILE=1records per-CTA phase timestamps and FC1 reduction-turn polling in a preplanned workspace tail for isolated single-tier decode. Compiled launch capacity checks and pool-mode invalidation protect the workspace contract. It is off by default and requires GPU qualification.Paired projection gathers can use posted-write push transport.
B12X_PCIE_DCP_A2A_PAIR_TRANSPORTselects pull or push independently; unset inherits the general DCP transport. Logical output widths skip trailing rank padding without changing valid columns.The fused paired-gather/router path can keep top-16 selection in registers. Scores, tie order, route weights, and their sequential FP32 normalization retain the scan path's contract.
Shape-selective fused MXFP8 activation quantization supports decode projections beyond the default output-width cap. Each shape needs a measured comparison before deployment.
B12X_W4A16_TOPK_SUM_OUTPUT=bf16orfp16rounds the full-rotation route sum once in the store. Compile keys, scratch, output buffers, and binding checks follow that dtype; unset retains FP32 output.B12X_W4A16_M8_CTA_THREADSselects 256 or 512 threads for the supported MoE geometry. The 512-thread option remains performance-unqualified in the DFlash2 composition.Collective tests cover graph replay, clipped outputs, and worker failure propagation. The QSRT extent benchmark records raw replay samples, operating state, FP32 diagnostic errors, and optional four-row launch attestation.
--server-metrics-urlmarks service overlap and leaves missing observations explicitly unknown.Offline verify-kernel qualification
Source
1414b128was compiled with CUDA 13.3.73, Torch 2.13.0 and CUTLASS DSL 4.6.2, without GPU device mapping. The four baseline CUDA.textsections match the archived production-contract objects byte for byte. Exact hashes are invalidation/performance/kimi_k3_verify_direct_pairs_offline_20260908.json.Lookup, FP8 conversion and HMMA counts stay constant; no local loads/stores appear. Three register increases remain flagged for device testing. Static instruction reductions are not latency measurements.
Validation of the added paths: 102 CPU collective tests and 60 targeted CPU MoE tests pass. The CPU-only run skipped nine GPU decoder tests; all nine subsequently passed on an RTX PRO 6000 Max-Q. Both nine-rank pull and push collective gates passed, including graph replay, raw 656-byte query records and output rows beyond 2 GiB. The summaries are retained in
validation/performance/kimi_k3_verify_gpu_correctness_20260908.json. All eight MoE width/CTA/pair combinations, an instrumented case, four TP9 gather specializations (pull/push × eager/graph), and six 2/3/4-bit × FP16/BF16 oracle specializations compile offline. The decoder oracles have executed successfully; the fused native-MoE variant and phase-record checks remain pending. Ruff still reports the base's undefined legacy helper inintrinsics.pyand twozip(strict=...)findings in_impl.py; changed standalone files and tests pass.Native-MoE equality, phase-record GPU validation, full-model evaluation and default decode-benchmark measurements for these changes remain pending.
benchmarks/benchmark_qsrt_verify_variants.pypreserves an interleaved native-extent harness with shared weights, mutated inputs and exact-object evidence; it is research-only and has not yet run on a device. Seedocs/dcp_query_output_layout.mdanddocs/qsrt_verify_diagnostics.mdfor contracts and qualification commands. The gather integration is in vLLM #654. The direct LUT consumer extends #328's modal-table decoder, and the stride ABI extends #335's transport; their existing work is retained in this PR rather than duplicated.Recorded device qualification (2026-09-07)
The paired-gather and routing checks recorded on nine idle RTX PRO 6000 GPUs on 2026-09-07 include 90 CPU collective tests, TP9 pull/push and captured-graph checks, and four GPU top-16 equivalence cases. Four-row paired-gather replay changes from 19.0 to 10.1 microseconds, and top-16 selection from 6.21 to 3.36 microseconds under that geometry. These operation-level results do not establish a universal model speedup. The fused-quantization shapes and output-store routes have separate compile, dtype, and binding tests; stale baseline binding tests are documented in the existing validation discussion.
The actual QSRT K2 source extents at logical TP9 ranks 0 and 4, widths 384 and 256, are preserved in the extent qualification document and its JSON record. All 16 cases across M=1/2/4/8 and 256/512 threads are finite, eager/graph identical, and have matching cross-thread output digests. Four-row traces attest the actual launch widths, registers, grid, and shared memory. The record identifies the frozen source pair and the limits of single-extent validation.
There is no qualified 512-thread speed comparison. Three of four replay intervals show live request activity. The rank-0 512-thread interval has idle endpoint gauges and unchanged counters, but its 256-thread reference overlaps serving. Raw timings remain diagnostic and are excluded from performance decisions. The routing/overlap-classification CPU suite passed eight tests, with Ruff and diff checks passing for that source.
The extent benchmark allocates output in the configured dtype, as implemented in revision
ab269462fcfc; instrumentation retains that contract. Packed MLA evidence belongs to B12X #311, and combined DFlash2 serving qualification belongs to vLLM #704. The invalid 128 Ki benchmark baseline is excluded; its request-ownership correction is llm-inference-bench #16.AI assistance was used for implementation, measurement, and evidence review, including GPT-5.6 Terra at maximum reasoning effort. Maintainer review is required before merge.