Repository navigation

M3 perf - #39466

Draft

kevin-mii wants to merge 155 commits into

sgl-project:mainfrom

kevin-mii:M3-perf

Draft

M3 perf#39466
kevin-mii wants to merge 155 commits into
sgl-project:mainfrom
kevin-mii:M3-perf

Commits on Sep 1, 2026
- MiniMax-M3: Gluon paged-attention sparse prefill with Triton fallback
  
  committed
  d2fd627
- MiniMax-M3: platform-gate the prefill kernel changes
  
  kevin-mii
  and
  claude
  committed
  2841575
Commits on Sep 3, 2026
- MiniMax-M3: rename SGLANG_OPT_USE_GLUON_PREFILL to SGLANG_MINIMAX_OPT_USE_GLUON_PREFILL
  
  zcnrex
  and
  claude
  committed
  4dbd95a
- Merge branch 'main' into m3-gluon-sparse-prefill
  zcnrex
  authored
  45d2702
Commits on Sep 4, 2026
- Merge commit '8a98f11078b0fc7beff6f7ce2b1ea59682d83171' into mm/m3-gluon-sparse-prefill
  kevin-mii
  committed
  47bcf53
Commits on Sep 7, 2026
- Merge commit '570087ceda91333c3fb4bad86dfbc28ffb4f4723' into r3/m3-gluon-sparse-prefill
  kevin-mii
  committed
  99ff340
- MiniMax-M3: allocate the lightning-indexer K cache in fp8 on ROCm
  
  zcnrex
  and
  kevin-mii
  committed
  d80cee9
- MiniMax-M3: account the fp8 index cache in the KV pool cell size
  
  kevin-mii
  and
  claude
  committed
  a709fcd
- MiniMax-M3: require gfx95 for the fp8 lightning-indexer K cache
  
  kevin-mii
  and
  claude
  committed
  9fe6bae
- kernels: wave64 histogram-select for the MiniMax-M3 decode top-k
  
  zcnrex
  and
  kevin-mii
  committed
  eb41ccb
- MiniMax-M3: keep the CUDA decode top-k ceiling at 4096 blocks
  
  kevin-mii
  and
  claude
  committed
  8c0b063
- MoE: single-launch small-batch sorting with fused mxfp8 activation quant
  
  zcnrex
  authored and
  kevin-mii
  committed
  e23731e
- MiniMax-M3: extend the fused MoE sort+quant to decode batches up to P=256
  
  kevin-mii
  and
  claude
  committed
  058d982
- MiniMax-M3: allow shared-experts fusion on ROCm gfx942 and newer
  
  authored and
  kevin-mii
  committed
  97a7a73
- MoE: gate the aiter small-sort patch to gfx950 and harden its install
  
  kevin-mii
  and
  claude
  committed
  9d8d5b5
- MiniMax-M3: gate shared-experts fusion with the arch helpers
  
  kevin-mii
  and
  claude
  committed
  0d84a54
- MiniMax-M3: MXFP8 dense-only block convert + torch._scaled_mm 1x32 path
  
  authored and
  kevin-mii
  committed
  6d08663
- MXFP8: platform-gate the dense block-convert env and the bf16 backend
  
  kevin-mii
  and
  claude
  committed
  be4ed6d
- MiniMax-M3: fuse add-RMSNorm with per-token FP8 quant, and the MoE all-reduce with it
  
  authored and
  kevin-mii
  committed
  ead758a
- MiniMax-M3: MFMA skinny fp8 PTPC GEMV for decode M=1 dense projections
  
  kevin-mii
  and
  claude
  committed
  1353757
- MiniMax-M3: keep the fused-quant path and all-reduce deferral ROCm-scoped
  
  kevin-mii
  and
  claude
  committed
  4cd5b8b
- MiniMax-M3: fix import order and move the skinny GEMV tests under the kernel taxonomy
  
  Kevin Mi
  and
  claude
  committed
  8c72a9a
- Merge branch 'main' into m3-gluon-sparse-prefill
  kevin-mii
  authored
  8b7acc7
Commits on Sep 8, 2026
- Merge branch 'main' into m3-gluon-sparse-prefill
  kevin-mii
  authored
  6d4f0ec
Commits on Sep 10, 2026
- Merge main into m3-gluon-sparse-prefill
  
  Kevin Mi
  and
  claude
  committed
  dba7ea2
- Merge remote-tracking branch 'upstream/main' into cl/36560
  Kevin Mi
  committed
  fab205e
- Merge remote-tracking branch 'upstream/main' into cl/36575
  Kevin Mi
  committed
  689ba00
- Merge remote-tracking branch 'upstream/main' into cl/36574
  Kevin Mi
  committed
  950005e
- Merge remote-tracking branch 'upstream/main' into cl/36559
  Kevin Mi
  committed
  e20cd80
- Merge main into m3-fp8-index-k-cache
  
  Kevin Mi
  and
  claude
  committed
  278e903
- Merge main into m3-rocm-shared-experts-fusion
  
  Kevin Mi
  and
  claude
  committed
  bd36cf2
- Name the fused allreduce result the way its siblings are named
  
  Kevin Mi
  and
  claude
  committed
  f6584b2
- Name the packed-page KV length ctx_len, not ctx
  
  Kevin Mi
  and
  claude
  committed
  d8927ae
- Carry the pending quant input in a ContextVar, not a module global
  
  Kevin Mi
  and
  claude
  committed
  06f671b
- Gate ROCm shared-experts fusion to gfx950, the only arch it was validated on
  
  Kevin Mi
  and
  claude
  committed
  b8fc6b0
Commits on Sep 11, 2026
Commits on Sep 12, 2026
- MXFP8 on gfx950: allow --moe-runner-backend aiter (FlyDSL fp8 per-1x32 fused MoE path)
  
  kevin-mii
  and
  claude
  committed
  c4e7f96
- MiniMax-M3 sparse attention: EAGLE chain target-verify on CUDA/ROCm
  
  kevin-mii
  and
  claude
  committed
  4c6ef3a
- MiniMax-M3 GPU chain verify: cross-layer top-k reuse and packed per-request index scoring
  
  kevin-mii
  and
  claude
  committed
  f38e76c
- MoE topk: do not emit the fused shared-expert marker twice on aiter paths
  
  kevin-mii
  and
  claude
  committed
  587c277
- MXFP8 gfx950: emit row-aligned scaled_mm operands from the quant kernel
  
  kevin-mii
  and
  claude
  committed
  0e7ce8a
- MiniMax-M3: fused scale+cast+scatter store for fp8 main and index KV caches
  
  kevin-mii
  and
  claude
  committed
  3aae177
- Triton attention: split-KV / grouped-head verify kernels for EAGLE draft extend and MiniMax-M3
  
  kevin-mii
  and
  claude
  committed
  09abe29
- MiniMax-M3 sparse attention: build eager verify metadata once per forward
  
  kevin-mii
  and
  claude
  committed
  0760f29
- Grouped-head verify kernel: long-context split budget and 16-head block for head_dim 128
  
  kevin-mii
  and
  claude
  committed
  011ea82
- MiniMax-M3: serve small extends over long cached prefixes with the decode-style kernels
  
  kevin-mii
  and
  claude
  committed
  3f5c2b3
- AMD: keep an explicit --triton-attention-num-kv-splits instead of pinning 16
  
  kevin-mii
  and
  claude
  committed
  65958e1
- Spec decode: token-block-parallel page-table copies in the Triton draft and EAGLE metadata paths
  
  kevin-mii
  and
  claude
  committed
  b4c2a64
- MiniMax-M3 sparse prefill: split the score-only index kernel over KV blocks
  
  kevin-mii
  and
  claude
  committed
  48d5beb
- Triton extend attention: split-prefix sweep with larger tiles for long cached prefixes
  
  kevin-mii
  and
  claude
  committed
  75a608d
- MiniMax-M3 Gluon prefill: build the page layout on the device
  
  kevin-mii
  and
  claude
  committed
  5dfd9bb
- MiniMax-M3 sparse attention: run as an eager break under breakable prefill graphs
  
  kevin-mii
  and
  claude
  committed
  e0da9eb
- Triton attention: serve grouped-head decode with the shared-KV verify kernel
  
  kevin-mii
  and
  claude
  committed
  1039c47
- MoE topk: let the Triton gate fill the fused shared-expert slot on aiter
  
  kevin-mii
  and
  claude
  committed
  09e90f6
- router_gemv: cover 65..128-row batches (EAGLE target-verify shapes)
  
  kevin-mii
  and
  claude
  committed
  24517d8
- OpenAI chat: log the validation error when a streaming request is rejected before its first chunk
  
  kevin-mii
  and
  claude
  committed
  d754dfb
- Quark: opt-in load-time FP8 for excluded (bf16) linear layers
  
  kevin-mii
  and
  claude
  committed
  0a36900
- Triton backend: serve large EXTEND chunks over a long prefix with aiter's paged batch-prefill
  
  kevin-mii
  and
  claude
  committed
  88cc1e1
Commits on Sep 13, 2026
- MiniMax-M3 on MI350X: status report (measurements, commits, configs)
  
  kevin-mii
  and
  claude
  committed
  bba0f3b
- MiniMax-M3 MI355X: launch, config and AIPerf AgentX benchmark scripts
  
  kevin-mii
  and
  claude
  committed
  536cea3
- MiniMax-M3 MI355X: status update (v9-v11 forced-acceptance points), lossy config uses 8192-token chunks
  
  kevin-mii
  and
  claude
  committed
  7736d0d
- Scheduler: chunked-prefill fairness for waiting extends (SGLANG_CHUNKED_PREFILL_FAIRNESS_RESERVE)
  
  kevin-mii
  and
  claude
  committed
  aa484ae
- MiniMax-M3 MI355X: configs pick up the chunked-prefill fairness reserve
  
  kevin-mii
  and
  claude
  committed
  08b152a
- MiniMax-M3 MI355X: final status report
  
  kevin-mii
  and
  claude
  committed
  9c90dec
- MiniMax-M3 MI355X: performance-only config also uses fp4 MoE activations at decode
  
  kevin-mii
  and
  claude
  committed
  389ee22
- MiniMax-M3 Gluon sparse prefill: dtype-aware, configurable gather-scratch cap
  
  kevin-mii
  and
  claude
  committed
  8645bbb
- MiniMax-M3 MI355X: final status with structural-lever results (EP, TP2xDP2, aiter glue, Gluon layout)
  
  kevin-mii
  and
  claude
  committed
  2cfe4e6
- FP8 per-token linear: consume the fused add-RMSNorm's pre-quantized activation (ROCm)
  
  kevin-mii
  and
  claude
  committed
  51e758e
- MiniMax-M3 MI355X: online PTPC-FP8 dense layers with tuned aiter a8w8 rows in the recommended config
  
  kevin-mii
  and
  claude
  committed
  52456aa
- MiniMax-M3 MI355X: status report with tuned PTPC-FP8 dense results
  
  kevin-mii
  and
  claude
  committed
  f16e994
Commits on Sep 14, 2026
- MiniMax-M3 sparse decode kernel: force each row's own local blocks in the packed scoring path
  
  kevin-mii
  and
  claude
  committed
  8095995
- MiniMax-M3 MI355X: PR index for the cleaned-up branches; configs set both extend-long-prefix env names
  
  kevin-mii
  and
  claude
  committed
  a92f604
- MiniMax-M3 MI355X: optimizations catalogue and refreshed PR index
  
  kevin-mii
  and
  claude
  committed
  ef761c4
- MiniMax-M3 MI355X: reproduction commands and images at the top of the optimizations catalogue; ship the tuned aiter rows
  
  kevin-mii
  and
  claude
  committed
  e3b73f3
- MiniMax-M3 MI355X: tighten the reproduction section; mori is not needed for this configuration
  
  kevin-mii
  and
  claude
  committed
  d16b641
- Trim the comments and docstrings in the chain-verify and page-table paths
  
  kevin-mii
  and
  claude
  committed
  97ae170
- Trim the long-context Triton attention comments to one-line constraints
  
  kevin-mii
  and
  claude
  committed
  c921bcb
- Cut the long-context Triton attention docstrings to their contract
  
  kevin-mii
  and
  claude
  committed
  f393992
- Port the fused-store validator, slot guards and cast-gated scales from the PR branch
  
  kevin-mii
  and
  claude
  committed
  6e7b377
- Own the aiter batch-prefill handle and descale tensors in the backend, not the module
  
  kevin-mii
  and
  claude
  committed
  ef087c8
- Name the extend-length lists in the Triton routing predicates by what they hold
  
  kevin-mii
  and
  claude
  committed
  07216c5
- Keep the long-prefix extend route out of graph capture
  
  kevin-mii
  and
  claude
  committed
  0e44aed
- Move the long-context attention kernel tests under test/registered/kernel/attention
  
  kevin-mii
  and
  claude
  committed
  8d41498
- Trim the R3 comments and docstrings to one line per decision
  
  kevin-mii
  and
  claude
  committed
  e544435
- Remove module-level state from the Gluon prefill path
  
  kevin-mii
  and
  claude
  committed
  8828a5b
- Guard the fused fp8 pair and the E8M0 scale memo against stale tensors
  
  kevin-mii
  and
  claude
  committed
  72b944f
- Derive the shared-slot fold from the JIT-gate dispatch predicate
  
  kevin-mii
  and
  claude
  committed
  6c5172a
- Move the R3 GPU tests to the registered-test taxonomy and drop the cap test
  
  kevin-mii
  and
  claude
  committed
  0b211b3
- Include the request id in the rejected-stream warning
  
  kevin-mii
  and
  claude
  committed
  a4b62b6
- style: isort import order in gluon_prefill
  
  kevin-mii
  and
  claude
  committed
  971468b
- Name the flattened-row metadata and buffers for what they hold
  
  kevin-mii
  and
  claude
  committed
  c461cd7
- Trim the comments and docstrings of the fused store, chunk fairness and online FP8
  
  kevin-mii
  and
  claude
  committed
  23fcb86
- Name the chunk-fairness and online-FP8 helpers by what they do
  
  kevin-mii
  and
  claude
  committed
  8dda12e
- Build the online-FP8 config as a cached property and name the envs in its log line
  
  kevin-mii
  and
  claude
  committed
  715b092
- Key the per-forward row-meta caches on id(forward_batch) like the extend cache
  
  kevin-mii
  and
  claude
  committed
  4997a4a
- Guard the two paths that assume their preconditions: pure safety, no-ops for the published config
  
  kevin-mii
  and
  claude
  committed
  0320cf8
- Rename the einsum subscripts in the aiter extend test: codespell reads thn as a typo
  
  kevin-mii
  and
  claude
  committed
  fab3d90
- Read the kv-splits value with getattr so non-ServerArgs stand-ins pass through the AMD hook
  
  kevin-mii
  and
  claude
  committed
  18a765a
- style: isort import order in triton_backend
  
  kevin-mii
  and
  claude
  committed
  664196e
- Drop the unused disable_value parameter of the small-extend row-meta builder
  
  kevin-mii
  and
  claude
  committed
  c7a80ce
- Tidy the session's environ.py comments to one line each
  
  kevin-mii
  and
  claude
  committed
  0d70689
- Prune the session's tests to the cases a plausible mistake fails, at the registry paths
  
  kevin-mii
  and
  claude
  committed
  cefeb81
- Trim the reproduction scripts: one-line comments, no measurement logs, no dead code
  
  kevin-mii
  and
  claude
  committed
  e3e11ef
- style: ruff/isort formatting of the reproduction scripts
  
  kevin-mii
  and
  claude
  committed
  01fb230
- Merge remote-tracking branch 'fork/M3-perf-cleanup-r3' into M3-perf
  kevin-mii
  committed
  5048659
- Merge remote-tracking branch 'fork/M3-perf-cleanup-r2' into M3-perf
  kevin-mii
  committed
  9f4a9f8
- Merge remote-tracking branch 'fork/M3-perf-cleanup-r1' into M3-perf
  kevin-mii
  committed
  8a389bd
- Trim the reproduction script headers and one quark comment to a single line under 100 columns
  
  kevin-mii
  and
  claude
  committed
  df226e6
- Gate the fused KV/index store on attributes the K-only index pool actually has
  
  kevin-mii
  and
  claude
  committed
  a1fe3d9
- MiniMax-M3 MI355X: PR index records the second integration pass and final branch heads
  
  kevin-mii
  and
  claude
  committed
  9f4454d
Commits on Sep 15, 2026
Commits on Sep 16, 2026
- MiniMax-M3 sparse prefill: sub-tile the KV block only on gfx950, where it was measured (mirrors PR 36546)
  
  kevin-mii
  and
  claude
  committed
  112539b
- MiniMax-M3 MI355X: PR index records the 36576 fix landing in the PR and the follow-up rebase state
  
  Kevin Mi
  and
  claude
  committed
  a5c2043
- MiniMax-M3 MI355X: PR index points at the one-line 36576 fix
  
  Kevin Mi
  and
  claude
  committed
  e2ab1a2
- MiniMax-M3 MI355X: PR index tracks the amended 36576 fix
  
  Kevin Mi
  and
  claude
  committed
  162fe45
Commits on Sep 17, 2026
- Follow PR 36546's rename to SGLANG_OPT_USE_MINIMAX_GLUON_PREFILL; launch scripts set both names
  
  Kevin Mi
  and
  claude
  committed
  dbfb4f0
- fix some m3 specific deployment issues
  
  zcnrex
  authored and
  kevin-mii
  committed
  3bbbc5c
- add some global fixes
  
  zcnrex
  authored and
  kevin-mii
  committed
  ef992de
- SGLANG_FORWARD_UNKNOWN_TOOLS make it surface calls for empty inventory
  
  zcnrex
  authored and
  kevin-mii
  committed
  c2d400a
- MiniMax-M3 endpoint hardening for the vendor format check
  
  kevin-mii
  and
  claude
  committed
  e185eec
- MiniMax-M3: expose set_dflash_layers_to_capture for DFLASH/DSPARK drafts
  
  kevin-mii
  and
  claude
  committed
  035888b
- SGLANG_ENABLE_QUEUE_FULL_429: answer a full waiting queue with 429 instead of 503
  
  kevin-mii
  and
  claude
  committed
  dd8fdd3
- MiniMax-M3 MI355X: external endpoint package (serve script, overlay Dockerfile, build runbook)
  
  kevin-mii
  and
  claude
  committed
  705bed5
- M3 endpoint: ruff format; INDEX_TOPK_FREQ override in serve_endpoint.sh
  
  kevin-mii
  and
  claude
  committed
  7bd5f70
- MiniMax-M3 tool parser: accept invoke tags with dropped or doubled name= glue
  
  kevin-mii
  and
  claude
  committed
  db61ee4
- M3 endpoint: numerics override knobs (QR_QUANT, KV_DTYPE, FP8_INDEX_CACHE, GLUON_PREFILL); handoff doc draft
  
  kevin-mii
  and
  claude
  committed
  442407e
- M3 endpoint: index top-k share frequency 1 by default
  
  kevin-mii
  and
  claude
  committed
  12753d2
- M3 endpoint: handoff document with the vendor-check results, the top-k bisection and the recommendation
  
  kevin-mii
  and
  claude
  committed
  5f94c88
- M3 endpoint: laptop hand-off for finishing the image push
  
  kevin-mii
  and
  claude
  committed
  2fa816d
- M3 endpoint: apply the aiter patch per file and assert the end state
  
  authored and
  kevin-mii
  committed
  9c92181
- M3 endpoint: drop serve_endpoint.sh for a plain `sglang serve` command
  
  Kevin Mi
  and
  claude
  committed
  052c854
- M3 endpoint: stop claiming NCCL_MIN_NCHANNELS was measured
  
  Kevin Mi
  and
  claude
  committed
  d91c401
Commits on Sep 18, 2026
- M3 endpoint: drop the 11 tuned-MoE rows the base image already ships
  
  Kevin Mi
  and
  claude
  committed
  2f50544

Commits

Commits on Sep 1, 2026

Commits on Sep 3, 2026

Commits on Sep 4, 2026

Commits on Sep 7, 2026

Commits on Sep 8, 2026

Commits on Sep 10, 2026

Commits on Sep 11, 2026

Commits on Sep 12, 2026

Commits on Sep 13, 2026

Commits on Sep 14, 2026

Commits on Sep 15, 2026

Commits on Sep 16, 2026