Repository navigation
Conversation
…rify kernels Kernels for MiniMax-M3.1-Flash on MI350X / MI355X with no callers yet: the NVFP4 sparse decode and Gluon prefill, the verify top-k indexer, QK-norm + RoPE, the NVFP4 KV and HiCache page-row paths, and the DSpark draft / accept kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… kernels - gluon_prefill: `_get_scratch` always allocates the scratch at bf16 width and only views it as fp8, so "256 MiB at fp8" and the grow step's "at bf16" misstated the footprint; the grow-step line is back to main's. - dspark kernel parity test: the HIP-only case list holds the TP-shard argmax as well as the step bias, so the comment names both. - verify_mla: the 13-26% speedup is a benchmark result for the PR body; the line only needs where num_stages=2 was measured. - index_topk_gfx950: reflow a three-line comment whose last line was a single word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"as in the Triton kernel" read as if a Triton NVFP4 decode kernel existed; the attention has only the Gluon kernel, and what it matches up to fp32 rounding is the bf16 Triton decode kernel run on the dequantized cache, the comparison test_minimax_nvfp4_sparse_decode makes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ke it `verify_topk_idx` and `_use_fused_index_topk` both branch on `isinstance(k_cache, Nvfp4Cache)`, but their signatures still said `torch.Tensor`; the sibling `_use_fused_verify_topk` already spells `torch.Tensor | Nvfp4Cache`. Annotation only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`scale_cast_to_cache` and `cache_inv_scale` are used only inside minimax_m3_qk_norm_rope.py (here and in every later branch of the series), and every other helper the fused cache kernel calls is `_`-prefixed; the public names suggested an API other modules import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`_gluon_sparse_prefill_group` ran `isinstance(k_cache, Nvfp4Cache)` for the KV-scale tensors and again three lines later for the scratch dtype; bind it once before both. `k_cache` is only rebound after the gather, below either use, so the two checks always agreed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- accept_greedy_target_predict looped over method-name strings and resolved them with getattr; iterate over `AcceptGreedy.torch` / `AcceptGreedy.triton` themselves, as the module docstring says every case does. - markov_step_bias imported `MarkovStepBias` midway through the case and re-bound it as `cls`; import it first, like the TP-shard argmax case, and drop the alias. Same inputs, same seeds, same assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ason test/README.md requires a file registered on CUDA plus another accelerator to carry a `backend-specific:` comment above the extra registration naming what only that backend catches. Here that is the gfx950-only step-bias and TP-shard argmax cases, which `_CASES` gates on `torch.version.hip`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… producer kernels Kernels for MiniMax-M3.1-Flash on MI350X / MI355X with no callers yet: MoE sorting + quant and the aiter MoE addend, the M3.1 small-M router, the split-K router GEMV with fused shared-expert SwiGLU, aiter's asm block-fp8 GEMM, the small-M fp8 projection, and the producers of their fp8 inputs: RMSNorm + quant epilogues and the TP4 decode boundary's fused all-reduce + residual add + RMSNorm + fp8 quant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hipcc.py: the 2-D grid comment named its caller (aiter_asm_blockscale); it
now states the accepted grid forms, which is what a reader of launch() needs.
- router_gemv.py: the fused partials + SwiGLU kernel passes partials_ptr as
out_ptr and counter_ptr and 0 as stride_om, which reads like a bug; say those
are never read without the fixup. Reword the `assert num_warps == 4` comment
("the two kernels' warps, so each keeps its layouts") so it reads as the
constraint: both kernels run at 4 warps alone, and sharing that count keeps
their layouts.
- m31_router.py: SHARED_N sat between the `_SHARED` table comment and the
table it describes; move it above the comment.
No behavior change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- moe_sorting_small module docstring: the fallback sentence ("Sorts this
module's kernels do not cover go to ..., larger ones that come with that
quant to ..., else to aiter") had to be parsed twice; state the three
destinations in order.
- presorted_moe_sort: the tuple layout sat inside the sentence and ran past
110 columns in an 88-column file; name the hand-off first, then the layout.
- router_gemv_partials_swiglu: "in one launch, bitwise:" did not say bitwise
what; say "bitwise equal to the two calls", as m31_shared_router does.
- test_build_is_cached_across_loads: the docstring told the incident (every
rank of every start loaded a fresh build); state the live constraint the
test guards instead, per the bug-regression docstring rule.
No behavior change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- moe_sorting_small.sorting_impl_wrapper: the presorted hand-off and both one-launch MXFP4 sorts each spelled out `plain_sort and expert_mask is None and num_local_tokens is None`; bind it once as `unmasked_sort`, so a reader sees the three paths share one eligibility. Read `_pending_fp4_input` once before the two one-launch branches instead of in each, and name the moe_sort_quant input after its MAX_QUANT_TOKENS cap `quant_input` instead of overwriting `fp4_input` with None. - m31_router: `8 if m <= 8 else 16` picked the router tile in three places (`_tile` and both `_SHARED` lookups); one `_token_bucket` helper keeps the `_TILES` and `_SHARED` keys from drifting apart. The conditions, their order and the values passed are unchanged: nothing between the hoisted ContextVar read and its old sites sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A JIT build of sgl-kernel's quick all-reduce, restructured for latency and cut to the INT4 / INT8 codecs at TP4 on bf16 data; same buffers, flags and arithmetic, so outputs are bitwise equal to sgl-kernel's. No callers yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t show The base header still said the copy serves the INT4 two-shot only; it serves INT8 as well. Its note that the grid cap comes from the loader repeats the QR_MAX_BLOCKS #error two lines below, so it goes. The entry-point header chained four clauses into one sentence; it keeps the provenance (sgl-kernel's qr_* calls) and the cut (INT4 / INT8, TP4, bf16 cast to fp16), and leaves the bitwise claim to the kernel headers that make it. raise_flags' comment restated its four-line body; the reason every wave waits for its own stores already sits on wait_own_stores. In the test, the grid-stride note now says which sizes exceed the grid, and the fallback cases note lists the INT4-at-TP2 case it left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The module docstring said the JIT build keeps sgl-kernel's buffer layout; the per-tile wire layout is wave-major now, and what it keeps is the buffer sizes, flags and arithmetic, as the kernel header says. It is now a summary line plus one paragraph. jit_serves' docstring ran three noun phrases together; it now names the argument and the two environment settings it reads. The fallback test's docstring named SGLANG_ROCM_QUICK_ALL_REDUCE_JIT, which this PR does not define (the integration PR adds it), and said a group raises at its first all-reduce, while TP2 would raise already at init. The bitwise test's docstring dropped the design note and keeps the contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the bf16 packed_* specializations used it, and the JIT copy dropped them with the bf16 kernels; nv_bfloat162 stays for the load/store casts. gfx950 device assembly of both codecs is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n one helper Both JIT workers defined the same nested open_comm and repeated the argument-order lambda, with the comment that explains it on only one of them. _open_comms now opens the pair in the same order (sgl-kernel first, then JIT), so each worker makes one call. The quant levels were a local 1, 2, 3 in one worker and bare 1 / 3 next to a named int6 in the other; both now read QuickReduceRegime, whose values QuickAllReduce passes as the level. The values are unchanged. No case was added or removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gfx950) Wires the gfx950 kernels into MiniMax-M3.1-Flash serving: the AMD model and MoE modules, the NVFP4 and Quark MXFP8 quantization paths, the sparse attention and Triton backends, DSpark on HIP, the overlap scheduler's seq-lens pull, the TP4 all-reduce fusions, the model overrides and aiter's tuned TP4 tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cut four comment blocks the PR added to the one fact each carries: - SGLANG_ROCM_QUICK_ALL_REDUCE_JIT: drop the bitwise-equality and unserved-group enumeration, which the quick_all_reduce_hip module docstring already states; keep the INT4 -> INT8 below 64 MiB fact a user flipping the variable needs. - _register_aiter_table: why aiter's model_configs/ tables stay out of the merge, in two lines instead of four. - quark_shared_experts_fusion_disable_reason: the two 3-line comments restated the reason strings returned right below them; keep only which source shape each branch detects. - BlockFp8CtxKvWriter._cache: rewrite the nested parenthetical so the cache layout and why the pool is held weakly read separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The override comment said the JIT build is bit-identical to sgl-kernel's, but this branch only runs when the level is INT4, and asked for INT4 the JIT build reduces everything below 64 MiB in INT8 (quick_all_reduce_hip's _INT8_BELOW_BYTES). It matches sgl-kernel bitwise at the level it actually runs, not at the level requested; say that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4 index cache Each TP rank scores a quarter of the index-K blocks for all four index heads and exchanges candidates over uncached IPC buffers; selects exactly what the per-rank indexer does. DSpark enables it only under static verify. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docstrings state the contract; measurements belong in the PR body. - dspark_hip module: drop "(25-77 MB a rank at 32-96 requests over xGMI)", the logits all-gather volume the deferred verify gather saves. - _m31_quark_gfx950_overrides: drop "(94 vs 206 tok/s at 8K input)"; the "halve batch-1 decode" fact that motivates forcing the block-fp8 conversion stays. - is_mxfp8_base_nvfp4_moe_source: summary plus one paragraph; the quant_method / moe_quant_algo keys are the code right below, and the NVFP4 tensor names live in remap_llm_compressor_nvfp4_expert. - mxfp8_linear_scale_alias: "load as before" described history; say a real `weight_scale` parameter keeps its name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three calls passed 9 to 13 arguments by position behind fmt: skip, and the NVFP4 scale expression appeared twice. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able find_spec returns a ModuleSpec or None, and a ModuleSpec always has an `origin` attribute (None for a namespace package), so getattr(spec, "origin", None) only hid a would-be AttributeError (.claude/rules/no-getattr-defensive.md). The None check and the falsy-origin early return are unchanged. Also build the table list as `[default, table] if default.is_file() else [table]` instead of filtering a one-element tuple through a comprehension; the resulting list is the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ationale The CP_MIN_VERIFY_REQS constants already say why short batches keep the native indexer; the comment now explains the 32-row tile limit instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test/README.md requires a `backend-specific:` comment above a registration that adds another accelerator to a CUDA-registered file, naming what only that backend can catch; the incremental admission check applies it to new or modified registered tests. This PR adds such a registration in three files: - swa/test_triton.py (MI35x suite): the gfx950 window-only DSpark draft verify in triton_backend_hip, skipped everywhere else. - kvcache/test_minimax_m31_hicache.py: the HIP build of the page-row transfer kernel, which the NVFP4 host pool now takes on ROCm. - dspark/test_dspark_minimax_mxfp8_stacked_kv.py: the gfx950 block-fp8 ctx-KV writer and fused markov step bias, skipped everywhere else. Comments only; the registrations are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep behind _cp_split_config ran on MI350X, not MI355X. Its request counts and context length move to the PR body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g to "Fit this image only" left the image unnamed. The indices come from ROCm 7.2's hipBLASLt; on ROCm 10 they fail verify graph capture with HIPBLAS_STATUS_INVALID_VALUE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sglang.srt.utils.network.get_open_port already binds a free port, and the sibling all-reduce test uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…lure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the tests that restate override defaults (Quark custom all-reduce, the Quark export's gfx950 defaults, the KV format default, the JIT quick all-reduce cutoff table, the gfx950 NVFP4 check, Blackwell attention TP), the scale-alias mirror, and the grouped-indexer gate mirror. The two fp8 tuple GEMM cases are one table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… merges Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
It changed FutureMap's publish, the scheduler's overlap step and the DSpark worker's decode for one gfx950 DSpark case, and measured +8.2% TPOT at one request, +2.8% at 8 and 32 (noise about 2.5%) and none at 64 on the old stack. The overlap path goes back to main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ables # Conflicts: # python/sglang/srt/environ.py
# Conflicts: # python/sglang/kernels/ops/moe/moe_fused_gate.py # test/registered/attention/unittests/swa/test_triton.py
…t window path Measured by tnguyeng on MI355X TP4 with amd/MiniMax-M3.1-Flash-Quark-MXFP4: +11.5% geomean output tok/s over 72k/120k at c32/c64, GSM8K 90.03% vs 89.88%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The module docstring repeated the PR description and environ.py; each row rule now sits on the function that applies it. DeepSeek-V4's module no longer restates its switches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fold the three bpreshuffle row cases into one synthetic table that also covers additions, and drop the row-count restatement of the DeepSeek-V4 CSV and the two real-aiter DeepSeek-V4 installs, which the all-tables real-aiter case covers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the pinned aiter, no superseded or missing row already means every row resolves to its own kernel, and a key aiter repeats counts as missing, so the per-row walk added nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the CPU checks of which Triton JSON applies and of the tile_m=16 policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or only The CP merge and the native verify now select through one helper, and test_minimax_verify_topk_idx already checks the fused and score + radix paths agree, so a second native comparison and four request shapes repeat that coverage. Keep the tied-score and short-row shapes. 134 -> 38 subtests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SGLANG_MINIMAX_M3_INDEXER_CP=1: the decode indexer's context partitioning now covers DSpark verify on the NVFP4 index cache. DSpark enables it only under static verify, since ragged verify trims rows per request._select_block_ids).Verification
acf8fdf93,SGLANG_USE_AITER=1), this branch:test_minimax_indexer_cp3 (38 subtests; the NVFP4 verify case compares with the fused selector only, since the CP merge and the native verify share one selection helper),test_minimax_indexer_cp_ipc1 (4 GPUs, exact against the native selector), CPUtest_minimax_indexer_cp_gate1 (7 subtests). pre-commit clean.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #42137432891
Latest PR Test (Extra): ❌ Run #42137432696
Latest PR Test (AMD ROCm 10): ⏳ Run #42137432607
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: