[liuhao-labs]

Motivation

SGLang supports I-quant GGUF weights for dequantization and MMVQ, but does not
currently include them in dense or batched-MoE MMQ dispatch. Beyond the
small-batch MMVQ boundary, dense I-quant prefill dequantizes the full weight to
BF16, while I-quant MoE prefill remains on the decode-oriented vector kernel.

This change adds the eight I-quant formats supported by the upstream MMQ
implementation. IQ1_M remains on fallback because that implementation does
not provide its MMQ dot product.

Fixes #35019

Modifications

• Add CUDA dense and batched-MoE MMQ support for IQ1_S, IQ2_XXS,
  IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL, and IQ4_XS.
• Apply a conservative M<=16 IQ MMQ gate. Wide matrices use MMVQ through M=8,
  MMQ for M=9..16, and dequantization from M=17. Narrow matrices retain MMVQ
  through M=16 and dequantization from M=17.
• Route pure or mixed IQ MoE to batched MMQ from M=128 only when assignment
  reuse, expert count, CUDA grid capacity, and both projection K alignments are
  suitable.
• Add an AOT capability query and enable IQ MMQ in SRT only when the installed
  sglang-kernel exposes that capability.
• Reject unsupported Q8_1 in the high-level selector and direct CUDA GGUF
  compute entry points.
• Harden shared MoE contracts with scalar catch-all registration, 64-bit expert
  strides, padded-tail bounds, complete q8 scale-tile loading, low-level
  alignment checks, and deterministic zero output for invalid or remote
  experts.
• Add registered CPU routing tests, CUDA dense and pure/mixed MoE regression
  coverage, and a checked-in dense benchmark using a revision-pinned GGUF
  fixture.

The SRT caller enables IQ MMQ only when its loaded kernel package exposes the
new capability; otherwise it retains the vector/dequantization fallback. The
historical compatibility tests below used sglang-kernel==0.4.6.post1.
At the current PR head and the latest tested main, the dependency pins have
advanced to sglang-kernel==0.4.9 and torch==2.14.1. A kernel release
containing the new capability and a follow-up pin update are still needed to
enable IQ MMQ in normal installations. This PR does not set an unreleased pin.

The kernel implementation is adapted from
aimbit-ni/vllm@037e1d547c15313aa7e2bb5fc83390ab0c857314 and the algorithms
in ggml-org/llama.cpp#8495. THIRDPARTYNOTICES.txt includes the ggml MIT
notice.

Accuracy Tests

Shared activation-policy revalidation, 2026-10-13

Latest tested main 7ff3a112846a3fcc12a27bba09d644885a226e4d (#43236)
enables --use_fast_math on Blackwell in the shared activation called by
GGUF MoE. I tested main and a clean detached merge with the unchanged PR head
0a39ec3a8754e3c4426950a5a93046c048a03f07; the integration tree is
36909c6658f0ce1ed346db5b85a3568325b9357d, still 15 files (+4127/-81).
The original PR branch was not rewritten or advanced for this verification.

Pinned report with raw logs, compiler invocations and reproduction scripts

• Separate cold-cache precise/fast-math builds: 60 activation comparisons on
  main and 60 on main+PR, all finite and within existing 0.01/0.01 bounds.
• Real two-projection GGUF + shared default JIT activation: 160 cases on each
  source, covering FP16/BF16, SiLU/GELU, eight IQ types, mixed Q5_0 directions
  and M=1/127/128/129. Fast-vs-precise max difference: 3.05176e-5.
• Registered routing CPU suite: 10 passed; registered activation subset:
  54 passed; existing GGUF AOT module: 153 passed. Four main+PR full MoE
  CUDA Graph replays match eager outputs exactly.

The component checks use Torch 2.11/CUDA13, newly compiled GGUF functions in an
isolated test namespace and the recorded auxiliary 0.4.4 wheel. They do not
prove the PR-built full wheel's ABI. No numerical kernel is mocked.

I then resolved the cached-image gap using the official hash-verified
sglang-kernel 0.4.9 wheel with matched Torch 2.14.1, Transformers 5.19.0,
FlashInfer 0.7.1, CUTLASS DSL 4.8.0 and Triton 3.8.0 in a separate environment.
The unmodified published wheel loaded successfully (SHA256
384a0d2158740f5ab42cfc64c872640bb3c80f3b0ec3b4666bc3efb553254bc4).
Both pinned main and main+PR loaded the exact prior 0.5B IQ3_S GGUF model and
returned health/generate HTTP 200, 8 prompt / 16 completion tokens, zero
cached tokens, and identical output token IDs/text. The canonical output hash
is 2604448b1bc1b2c9a10961f6b562bed70aab5041db4cb8a7db54e83f9cfde287.
Paired serving provenance, raw logs and comparison
No test-namespace operator replacement or version-check bypass was used in
these serving runs; capability is false with the published wheel.

This proves the current released-wheel small dense-model fallback/serving path,
not enabled IQ MMQ full-model serving or the PR-built full wheel's ABI. Official
CI and H100/B200 remain separate requirements. The first 0.03 memory-fraction
attempt hit the real KV budget guard, so both final arms used 0.05, the same
1024-token cap, one request and a 10 GiB container limit. No PR-related
numerical regression or required feature-code change was found.

Published MoE binding follow-up, 2026-10-13

I additionally tested latest main
27b68303d043ddbddf9861c55c26bc8617ea7c35 and its clean detached merge with
this PR (449c32e493e80e67190fcf1f61976dc6045159e9, tree
bb5c83cc334e80c51e6a0eb44045a5358c189162) using the unmodified official
0.4.9 production common_ops and Torch 2.14.1/CUDA13. Each source passed
32 real two-projection expert-layer cases: eight IQ formats, FP16/BF16,
SiLU/GELU, M=128/E=8/top-k=2, input/intermediate width 256. Outputs were
finite and within atol=1, rtol=0.1 of the dequantized reference; maximum
absolute error was 0.00366211. All 32 paired outputs were bit-identical.
Profiler records two real sgl_kernel::ggml_moe_a8_vec calls and the shared
default fast-math activation; fresh caches and actual compiler flags were
recorded. Capability remains false, so this is released-wheel expert-layer
fallback validation, not enabled IQ MMQ or full expert-model serving.

Raw outputs, library hashes, compiler commands and build attempts

I attempted the complete current-source AOT wheel with matched Torch and
one build worker/NVCC thread. Default builds hit cgroup OOM at 10 and 14 GiB
in existing FP8 units; an all-source/all-default-target build with SM121a-only
common_ops also hit 14 GiB OOM in the same existing fp8_gemm_kernel.cu.
Those failing source units are identical on latest main and main+PR.
Commands, exit 137, three Docker OOM events and resource samples are included.
No wheel was emitted: the prepared 80 enabled-production cases and 153-test
GGUF run against a new full wheel were not executed. New full-wheel ABI,
capability=true expert validation and official H100/B200 CI remain outstanding.
The earlier component/test-namespace passes do not fill these gaps.

The 15 upstream commits since the earlier frozen main did not change GGUF,
shared activation/JIT core or AOT build metadata by blob. The clean merge
still has the same 15-file feature diff; the original PR head was not rewritten.
Latest main and official 0.4.9 still lack IQ-MMQ capability, so the proposed
support remains necessary. This follow-up is correctness/build evidence;
historical speed figures below retain their original pinned runtime.

Historical pinned-runtime validation

Local validation used the installed CUDA 13/aarch64 AOT wheel on one NVIDIA
GB10 (sm_121), rather than JIT-compiled kernels.

The prior validation used upstream base
0c84eaba7f8d6674a360839426cacde4e1acd851 and clean local head
4c3e4c2350f4d6c4ff2a785e9b294938766a3cc6. On 2026-10-09, upstream main
was synchronized through 6e7beace143964386ce2e4418401b7369f0dabd0 into the
current PR head 0a39ec3a8754e3c4426950a5a93046c048a03f07. The three-way merge
was clean and the feature diff remains exactly 15 files (+4127/-81). The 67
intervening main commits have no path overlap with the feature files; the four
overlapping paths from the earlier sync contained only AOT cleanup and
formatting, not GGUF MMQ logic. The local numbers below belong to the historical
validation head and runtime; they do not validate the current upstream runtime
end to end. The refreshed head is pending official CI.

• historical validated wheel SHA-256:
  626d6ee79dd37dae2242f2f18a97b0536c95dfa4c053cb56d7573de23324c6e0
• registered CPU routing and compatibility suite: 10 passed
• patch-focused GGUF CUDA matrix: 57 passed
• complete python/sglang/kernels/aot/tests/test_gguf.py:
  153 passed
• installed-wheel production smoke: passed

Using the same CUDA 13/aarch64 build configuration, the compressed wheel grew
from 29.473 MiB to 30.999 MiB: +1.527 MiB (+5.18%). Each bundled
architecture-specific common_ops shared object grew by 917,504 bytes.

The historical wheel was built before the August main-only merge. Its AOT tree,
SRT GGUF selector and registered routing test matched the August validation head
4c3e4c2350f4d6c4ff2a785e9b294938766a3cc6. That source-equivalence statement
does not extend to the October runtime or today's shared activation policy.

The production smoke observed the new capability as enabled and covered:

• wide dense IQ MMQ at M=16 and dequantization fallback at M=17
• exact fixed-shape CUDA Graph capture/replay
• MoE vector dispatch at M=127 and batched dispatch at M=128
• pure and mixed quantization at M=128, M=256, and M=2050
• Q8_1 rejection, K-alignment guards, and invalid-expert zero output

Maximum reserved GPU memory during the production smoke was 528 MiB.

A compatibility smoke combined the patched SRT source with a locally cached
sglang-kernel==0.4.6.post1 baseline artifact (SHA-256
c7aac3ab20ff3c704e2bbc1a2fa78e5142a65150f12bca74f8e4cbd4f651d958). This is
not the published PyPI artifact. The published aarch64 wheel has SHA-256
7eccd61408b44b55ab35cdbcf8a05243ce2a9a5e781fabb00da375af7d7130a1; the
external 4k run below used that published wheel. The local smoke observed
capability=False, selected dequantize for an IQ dense input, and returned
shape [9, 18944]. This verifies that the combined source remains correct
before the new kernel wheel is released.

A final-head bounded end-to-end serving sanity check also passed with the
artifacts in /tmp/sglang-36122-e2e-final.XwQYAu and the revision-pinned
legraphista/Qwen2.5-0.5B-Instruct-IMat-GGUF
Qwen2.5-0.5B-Instruct.IQ3_S.gguf file (SHA-256
b7685f565fc6e49a8f9bc0fe09b9bb22ca0d5e4b10635cd4b181597f113794b6).
The installed patch wheel (SHA-256 626d6ee7...) reported capability=True;
the same synced SRT with the local baseline artifact (SHA-256 c7aac3ab...)
reported capability=False. Both configurations loaded the model, returned
HTTP 200 from the health and generation requests, and recorded zero cached
tokens for generation. Their client JSON files were byte-for-byte identical
with SHA-256
bd3c11d390f7a373e534f5361807119b2c040bc325cfe1f6fd2c8165d4426933:
11 prompt tokens, 16 completion tokens, a length finish, and text:

 I like to work outside because it is a great way to get exercise. I

Both runs used a single GB10 (sm_121, CUDA 13, aarch64), read-only source,
wheel, and model mounts, no network or host port, a 10 GiB container memory
limit, --mem-fraction-static 0.03, --max-total-tokens 1024, one running
request, Triton attention, PyTorch sampling, and CUDA graphs disabled. The patch
run loaded 0.88 GB of weights and allocated approximately 0.02 GB of KV cache.
This validates a small dense IQ3_S model and the released-wheel fallback; it
does not replace testing the original approximately 100 GB routed-expert model.
The critical server-log scan found no traceback, CUDA error, NaN, or
illegal-memory failure; the remaining messages were the expected nvidia-smi
fallback and shared local-tokenizer warnings.

The H-specific Lint workflow
passed, including the repository pre-commit checks, documentation checks, and
the dedicated sgl-kernel clang-format check. The older
Lint workflow
also passed for prior head 0137a1e.... Normal platform CI is gated on the
repository's run-ci label.

The complete local AOT collection is not claimed as passing. A bounded run
reached 971 passed and 2 skipped before an unrelated large AWQ parameter
exceeded the allocator cap. Excluding AWQ reached 970 passed and 2,310 skipped
before FlashMLA sparse prefill rejected SM121.

Normal/full PR CI is still pending because the repository's required run-ci
label has not yet been added. The current red PR Test entries are gate failures
for that missing label, and their wheel, H100, and B200 jobs were skipped; they
are not kernel test failures. A maintainer must add run-ci (or run the
project's authorized slash command) so the normal workflow builds and installs
the PR-local wheel on the official x86_64/aarch64 and H100/B200 environments.

The Issue reporter subsequently completed a frozen 4k cold-prefill comparison
on this H-pinned source using a reconstructed approximately 99.18 GiB
DeepSeek-V4-Flash 0731 checkpoint (report,
follow-up).
The checkpoint is a reconstruction from surviving Unsloth shards rather than a
bit-identical copy of the deleted original: two MXFP4 expert tensors were
converted to Q8_0 and the remaining tensors were byte-copied. The H source with
the published released aarch64 wheel selected the vector MoE path, while H with
the H-built wheel selected batched MMQ at M>=128; both arms used the same H
source and passed the route and correctness checks.

Across three measured 4k runs (concurrency 1, cold prefill), prefill throughput
was 60.93 versus 97.37 tokens/s (+59.8%), TTFT was 26,513 versus 15,912 ms
(-40.0%), and decode TPOT was 61.60 versus 62.09 ms (+0.8%). The literal U plus
released-wheel arm was not run separately; the reporter's historical 0.5.18
baseline reproduced the issue-scale result. A 16k run was not feasible under
the frozen approximately 99 GiB model and 121 GiB unified-memory budget because
the 4k run left only about 2-3 GiB for KV growth. The reconstructed model also
required an out-of-tree loader patch, so these numbers are targeted external
validation, not an official full-main serving result.

Speed Tests and Profiling

These are directional route-selection microbenchmarks from a shared GB10 that
retained a live Qwen service. They are not uncontended end-to-end throughput or
cross-architecture claims. They predate the main sync and were not rerun; the
sync did not change the IQ kernel patch represented by these measurements.

The checked-in benchmark compares direct MMQ with dequantization plus BF16
matmul using revision-pinned 18944x1024 GGUF fixtures. The table below uses
two independent benchmark invocations. Each value is the mean speedup, with the
two-run range in brackets; speedup is (dequantize + BF16 matmul) / MMQ.

quant type	M=16 speedup, mean [range]
IQ1_S	1.75x [1.75x, 1.76x]
IQ2_XXS	1.31x [1.29x, 1.33x]
IQ2_XS	1.26x [1.26x, 1.26x]
IQ2_S	1.60x [1.59x, 1.61x]
IQ3_XXS	1.71x [1.67x, 1.74x]
IQ3_S	1.66x [1.65x, 1.67x]
IQ4_NL	1.48x [1.47x, 1.48x]
IQ4_XS	1.26x [1.24x, 1.28x]

The full M=9/16/17/24/32 matrix and reproduction notes are in the
benchmark comment.
The M=16 ranges retain at least 1.24x measured headroom across all eight
formats. Although the isolated M=17 sample was also faster, M=18..23 were not
measured and M=24 regressed on multiple formats. The shared GB10 measurements
do not justify interpolating a larger universal CUDA gate, so this PR uses the
conservative M<=16 boundary. M=17 and above retain dequantization.

Directional MoE measurements were:

path	vector baseline	batched MMQ	speedup
IQ3_S/Q5_0 M=128	2.249 ms	1.490 ms	1.51x
Q5_0/IQ3_S M=128	2.524 ms	1.571 ms	1.61x
IQ3_S/Q5_0 M=256	4.497 ms	1.758 ms	2.56x
Q5_0/IQ3_S M=256	5.145 ms	2.283 ms	2.25x
IQ3_S/Q5_0 M=2050	34.801 ms	9.056 ms	3.84x

The external 4k result above is a targeted serving throughput validation; it is
not a claim about uncontended throughput or a literal bit-identical checkpoint.
The 16k comparison and a separately measured literal-U arm remain unavailable
under the reported hardware/model constraints. Official x86_64/aarch64,
H100/B200 CI and maintainer review are still pending; see the
original reviewer response
and the
pinned U/H protocol.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations. (N/A: this changes internal CUDA dispatch without adding a user-facing flag or supported format; the in-tree benchmark and third-party notice were updated.)
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #37882097010
Latest PR Test (Extra): ❌ Run #37882096734
Latest PR Test (AMD ROCm 10): ❌ Run #37882097020

[liuhao-labs]

GB10 dense IQ MMQ crossover

This plot uses the revision-pinned 18944x1024 GGUF fixtures on CUDA 13 / GB10
(sm_121). Each cell is the mean speedup from two independent invocations of
the checked-in benchmark, with the two-run range shown below it. Speedup is
(dequantize + BF16 matmul) / MMQ.

At the sampled MMQ boundary M=16, the two-run ranges retain at least 1.24x
measured headroom across all eight formats. The isolated M=17 sample was also
faster, but M=18..23 were not measured and M=24 regressed on multiple formats.
To avoid interpolating a universal CUDA threshold from one shared GB10, the PR
uses the conservative M<=16 gate; M=17 and above remain on dequantization.

Run the underlying benchmark twice from python/sglang/kernels/aot in an
environment with the current PR wheel installed. The --local-files-only flag
requires the pinned fixtures to already be present in the Hugging Face cache;
omit it on the first run if the environment has network access.

for run in 1 2; do
  python benchmark/bench_gguf_mmq.py \
    --batch-sizes 9,16,17,24,32 \
    --shape 18944x1024 \
    --local-files-only | tee "/tmp/gguf-mmq-run-${run}.md"
done

The heatmap was rendered from those two benchmark tables; the plotting-only
helper is not part of this PR. These are directional local microbenchmarks on a
shared GB10 with a live Qwen service, not uncontended end-to-end throughput or
cross-architecture results.

[liuhao-labs]

Additional end-to-end compatibility check

I ran the revision-pinned
legraphista/Qwen2.5-0.5B-Instruct-IMat-GGUF
Qwen2.5-0.5B-Instruct.IQ3_S.gguf file (SHA-256
b7685f565fc6e49a8f9bc0fe09b9bb22ca0d5e4b10635cd4b181597f113794b6)
through SGLang's real /generate endpoint on a GB10 (sm_121, CUDA 13,
aarch64).

configuration	capability	prompt/completion tokens	result
patched SRT + PR-local wheel (dae0fcc2...)	true	11 / 16	pass
patched SRT + released sglang-kernel==0.4.6.post1 (c7aac3ab...)	false	11 / 16	pass

Both greedy requests produced byte-for-byte identical output:

 I like to work outside because it is a great way to get exercise. I

The runs used read-only source/wheel/model mounts, no network or host port, a
10 GiB container limit, --mem-fraction-static 0.03,
--max-total-tokens 1024, one running request, Triton attention, PyTorch
sampling, and disabled CUDA graphs. The patch run loaded 0.88 GB of weights and
allocated approximately 0.02 GB of KV cache.

This is a small dense IQ3_S serving and released-wheel fallback check. It does
not claim end-to-end coverage of the original approximately 100 GB
routed-expert model; pure and mixed MoE paths are covered by the focused CUDA
and installed-wheel production tests described in the PR body.

[liuhao-labs]

Refreshed-head validation

The Draft branch is now synchronized and pinned to:

• U = 0c84eaba7f8d6674a360839426cacde4e1acd851 — upstream main snapshot used as the validation baseline
• H = 4c3e4c2350f4d6c4ff2a785e9b294938766a3cc6 — current Draft PR head; this merge commit contains U

The feature diff from U to H remains 15 files (+4127/-81). The CUDA 13/aarch64 wheel SHA-256 is 626d6ee79dd37dae2242f2f18a97b0536c95dfa4c053cb56d7573de23324c6e0 (30.999 MiB). The wheel-build head and H have identical AOT tree, SRT GGUF selector, and registered routing-test Git objects; the final merge only incorporated upstream changes outside those validated surfaces.

Validation on one GB10 (sm_121, CUDA 13, aarch64):

• registered CPU routing/compatibility: 10 passed
• focused GGUF CUDA matrix: 57 passed, 5 warnings in 49.07s
• complete test_gguf.py: 153 passed, 5 warnings in 60.32s
• installed-wheel production smoke: passed with a 0.06 allocator cap (528 MiB maximum reserved)
  • dense M=16 -> sgl_kernel::ggml_mul_mat_a8
  • dense M=17 -> sgl_kernel::ggml_dequantize
  • MoE M=127 -> sgl_kernel::ggml_moe_a8_vec
  • MoE M=128 and M=2050 -> sgl_kernel::ggml_moe_a8
  • fixed-shape CUDA Graph capture/replay matched exactly

I also reran the small revision-pinned IQ3_S serving check on H with two kernel-wheel pairings:

pairing	capability	health/generate	cached prompt tokens	result
H source + refreshed PR wheel (626d6ee7...)	true	200 / 200	0	pass
H source + released sglang-kernel==0.4.6.post1 (c7aac3ab...)	false	200 / 200	0	pass

The two client JSON files were byte-for-byte identical (SHA-256 bd3c11d390f7a373e534f5361807119b2c040bc325cfe1f6fd2c8165d4426933): 11 prompt tokens, 16 completion tokens, length finish, and identical greedy text. The critical server-log scan found no traceback, CUDA error, NaN, OOM, or illegal-memory failure.

This remains a Draft. The original approximately 100 GB routed-expert 4k/16k A/B, official H100/B200 AOT runs, and official x86_64/aarch64 wheel CI are still pending.

[GaelicThunder]

Thanks @liuhao-labs — the pinned-protocol run is done, on the exact setup from #35019: same GB10 box (sm_121a, CUDA 13, aarch64), same 100 GiB DeepSeek-V4-Flash 0731 GGUF recipe, same harness.

Pins and gates — verified

• Test source git rev-parse HEAD = 4c3e4c2350f4d6c4ff2a785e9b294938766a3cc6 (H), matches the current Draft tip.
• git merge-base --is-ancestor U H passes — H contains U.
• Feature diff U..H = 15 files, +4127/−81, as pinned.

Model

The exact GGUF behind the issue's numbers had been deleted from disk. I reconstructed it from the surviving Unsloth UD-IQ3_XXS shards: the two MXFP4 routed-expert tensors (blk.26/blk.42 .ffn_down_exps.weight) converted to Q8_0, every other tensor copied byte-for-byte. Verified tensor histogram matches the issue recipe exactly:

F32 662 · Q8_0 323 · Q6_K 170 · IQ3_XXS 75 · IQ2_XS 50 · BF16 43 · IQ3_S 2 · I32 3
1328 tensors · 99.18 GiB · sha256 17f57289e0d41f913ada93132b645e0920029391f1b3d973b3e5356840c49988

All 127 I-quant tensors are routed-expert tensors (blk.N.ffn_{gate,up,down}_exps); every dense tensor is Q6_K/Q8_0/BF16.

Arms

Arms 2 and 3 share an identical SGLang source tree — only the kernel wheel differs.

arm	source	rev-parse HEAD	kernel wheel	IQ MMQ capability
baseline repro	original issue image sglang 0.5.18.dev564+gbe804c1b8 + its released kernel	—	released	symbol absent
2 (core)	H	4c3e4c2350f4…	released 0.4.6.post1 (aarch64, 7eccd614…)	absent → False
3 (core)	H	4c3e4c2350f4…	built from H checkout (95cd2ccd…, 32 521 894 B / 31.01 MiB)	True

Arm 1 (U + released wheel) was not run as a separate arm: the 0.5.18 baseline reproduction above fills the operational-baseline role against the published issue numbers. If you want the literal U-arm as well I can add it, but it does not change the kernel comparison — arms 2 vs 3 are the isolated pair.

Route evidence (not the capability flag)

The capability boolean is insufficient on its own, so the PR's own dispatch functions were called from the installed wheel against the installed source at real model shapes (hidden 4096, moe_intermediate 2048, E=256, top_k=6, IQ3_XXS):

Arm 2 — H + released wheel:

ggml_supports_iq_mmq   -> ABSENT (ImportError)
_supports_iq_mmq        = False
IQ3_XXS in MMQ_QUANT_TYPES = False
MoE   M=1..16384       -> ggml_moe_a8_vec (per-token vec)
dense M>=17            -> dequantize

Arm 3 — H + wheel built from H:

ggml_supports_iq_mmq() -> True
_supports_iq_mmq        = True
IQ3_XXS in MMQ_QUANT_TYPES = True
MoE   M=1,8,64,127     -> ggml_moe_a8_vec
      M=128            -> ggml_moe_a8 (batched MMQ)
      M=129,4096,16384 -> ggml_moe_a8
dense M=16             -> MMQ ; M>=17 -> dequantize

So ggml_moe_a8_vec for arm 2 and ggml_moe_a8 for arm 3, and the IQ_MOE_MMQ_MIN_TOKENS = 128 gate flips exactly where the PR says it does.

One scoping observation: issue #35019 named two fallbacks — dense projections and MoE experts. This PR closes the MoE one. The dense path still routes to dequantize for M >= 17 (dense MMQ gate is M <= 16), so prefill-sized batches never reach it. For this checkpoint that is nearly moot — all I-quant tensors are routed experts and all dense tensors are Q6_K/Q8_0/BF16 — but I want the issue not to be read as fully closed on the dense side.

Measured — ctx 4k, frozen argv, cold prefill

Frozen argv, identical for every arm, only the kernel wheel differs:

--model-path <gguf> --port 30099 --host 127.0.0.1 --context-length 8192
--mem-fraction-static 0.93 --max-running-requests 1 --trust-remote-code
--cuda-graph-max-bs 2 --moe-runner-backend triton --disable-radix-cache

with SGLANG_GGUF_SYNC_LOAD=1. Bench: --dataset-name random --random-input-len 4096 --random-output-len 64 --num-prompts 10 --max-concurrency 1, one unmeasured warmup then three measured runs, fresh server per arm incl. --disable-radix-cache and unique random prompts per measured request.

metric	arm 2 — H + released wheel	arm 3 — H + wheel built from H	delta
prefill tok/s med [min–max]	60.93 [60.91–62.31]	97.37 [96.29–98.70]	+59.8 %
median TTFT (ms)	26 513 [26 191–44 070]	15 912 [15 707–26 409]	−40 %
TPOT (ms)	61.60 [61.59–61.67]	62.09 [62.06–62.18]	+0.8 %
decode tok/s	16.23	16.11	−0.7 %
min free unified memory	2.06 GiB	3.06 GiB	+1.0 GiB

Per-run (prefill / median TTFT / TPOT):

arm 2   60.93  44069.62  61.60      arm 3   97.37  26408.89  62.09
        62.31  26191.11  61.59              98.70  15707.18  62.06
        60.91  26513.34  61.67              96.29  15912.04  62.18

The decode column is the control. Decode runs at M = 1, below the IQ_MOE_MMQ_MIN_TOKENS = 128 gate, so it must not move — and it does not: TPOT shifts 0.8 %, inside run-to-run noise, while prefill moves 60 %. That isolates the gain to the batched MoE kernel rather than anything environmental. Prefill spread within each arm is 2.3 % (arm 2) and 2.5 % (arm 3); the two distributions do not overlap.

Arm 3 also uses about 1 GiB less memory at the same workload, consistent with MMQ consuming the quantised weights directly instead of materialising them. On a box where this model leaves only a couple of GiB of headroom, that matters as much as the throughput.

Correctness, temperature 0, both arms — coherent, no NaN or garbage:

"The sky is blue because sunlight is scattered by tiny mole..."
usage: {'prompt_tokens': 10, 'completion_tokens': 60}

Baseline reproduction (control on the reconstructed model)

The original issue image sglang 0.5.18.dev564+gbe804c1b8 + released kernel against the reconstructed GGUF at ctx 4k:

metric	this run	issue #35019
prefill (input tok/s)	57.69	56.3
decode (from TPOT 61.53 ms)	16.25 tok/s	16.78 / 16.36
median TTFT	20 373.93 ms	—

Prefill within 2.5 % and decode within 3 % of the published figures — so the reconstructed checkpoint and this box both reproduce the issue's baseline. That is the control the arm numbers hang off.

Notes for reproduction

1. The pinned released-wheel sha256 c7aac3ab… does not exist on PyPI. sglang-kernel==0.4.6.post1 publishes exactly two files: …manylinux2014_aarch64.whl 7eccd61408b44b55ab35cdbcf8a05243ce2a9a5e781fabb00da375af7d7130a1 (30 741 582 B) and …manylinux2014_x86_64.whl 5fee1007… (377 094 734 B). Worth correcting the pin in the PR body/validation comment so nobody chases a hash that doesn't match a published artifact.
2. Build the kernel wheel against the torch SGLang pins. H pins torch==2.13.0; a wheel built against 2.11 installs and imports fine, then fails at common_ops load with undefined symbol: _ZNK2at10TensorBase14const_data_ptrIiLi0EEEPKT_v. Two further build/run mismatches cost hours here, both same class: building against torch 2.13 and running on 2.11 → undefined symbol: _ZNR5torch7Library4_def...; building in a container without --gpus → undefined symbol: cuPointerGetAttribute. Both surface as No module named 'common_ops', which hides the real cause.
3. DeepSeek-V4 GGUF does not load on stock SGLang. _get_gguf_weights_map raises RuntimeError: Unknown gguf model_type: deepseek_v4; only the loader accepting deepseek4 (expert_pack, merged Support deepseek v4 and kimi k3 on ssd #35314) routes routed experts through mxfp4_matvec, never the ggml MMQ kernels. The original setup carries an out-of-tree patch stack applied identically to both arms. Worth knowing if this is ever meant to run in CI as-is.
4. --mem-fraction-static 0.03 is not viable for this model on 121 GiB unified: the engine profiler refuses it (Loaded weights leave no GPU memory for the KV cache … Raise above 0.914). Frozen at 0.93 — the value the original harness used.
5. Current-main runtime cannot serve this checkpoint on this box. sglang 0.5.19 + torch 2.13 drains unified memory from ~9 GiB to under 1 GiB within ~15 s of the server reporting ready, before the first benchmark request produces a prefill batch, then the machine kernel-OOMs (NVRM: Out of memory [NV_ERR_NO_MEMORY] in kmsg). Identical on both arms, so unrelated to this PR. The same model, same argv, under the 0.5.18-derived container with H installed (self-reports 0.5.19.dev440+g4c3e4c235 torch 2.11.0) runs with 2–3 GiB to spare, so all arm numbers were taken there, deliberately. Worth flagging: the large-model validation this PR is waiting on may not be runnable on a pristine current-main environment.

16k — not run, reason documented

The 4k arms already sit at 2–3 GiB of free unified memory; a 16k KV pool does not fit alongside ~99 GiB of weights on this box under the frozen memory constraints, so ctx 16k was not attempted rather than misreported. If you want the 16k column anyway, I can attempt it with the same arms and a documented memory trade-off — just say the word.

Preliminary: prefill 4k improves +59.8%, TTFT −40%, decode flat, peak-memory headroom +1 GiB, on identical args, cold prefill, route-verified. This is exactly the class of result I hoped this PR would produce; the conservative gating and the capability handshake make the separation between arms clean.

[GaelicThunder]

Quick follow-up, separate from the data: thanks for the pinned protocol — it's what made this run interpretable, and that's genuinely rare. Arms 2 vs 3 differing by exactly one wheel, with the route selection visible in the logs rather than hidden behind a capability flag, is the shape that lets a result stand on its own instead of on trust. I'm around if the 16k column is wanted.

[liuhao-labs]

Thanks for the H-pinned 4k validation in the latest result.

I have updated the PR body with the reconstructed-checkpoint caveat and the published aarch64 wheel hash. The result is:

• H + released wheel: vector MoE path, 60.93 tok/s prefill
• H + H-built wheel: batched MMQ from M>=128, 97.37 tok/s prefill (+59.8%)
• TTFT improved from 26,513 ms to 15,912 ms; decode TPOT changed by +0.8%

The literal U arm was not run separately, and 16k does not fit the reported 121 GiB unified-memory budget after the approximately 99 GiB reconstructed checkpoint. Those limitations are now explicit in the body. The reconstructed checkpoint is targeted evidence, not a bit-identical copy of the deleted original.

The PR is now Ready for review and CODEOWNERS have been requested. Could a maintainer please review the AOT/SRT changes and trigger the normal platform CI when convenient? The earlier red checks were Draft-gate failures; the H-specific lint workflow passed.

[liuhao-labs]

/tag-and-rerun-ci

[liuhao-labs]

@BBuf @merrymercy @HaiShaw @ispobock

The PR is now Ready for review, and the H-specific lint check is green. The issue reporter has completed the H-pinned 4k external A/B (+59.8% prefill, -40% TTFT; decode control stable); the reconstructed-checkpoint and 16k memory caveats are documented in the body.

Could a maintainer please review the AOT/SRT changes and add the run-ci label (or run /tag-and-rerun-ci)? The current red checks are only the new gate's missing-label failures; the actual platform jobs have not started yet. The requested CODEOWNER list is populated automatically.

[liuhao-labs]

The current red checks are still stopping in pr-gate before any build or test job starts. The gate reports PR Draft: false, labels [sgl-kernel], then Missing required label 'run-ci'; downstream jobs are skipped.

I also tried adding run-ci through the GitHub API, but the repository returned 403 Must have admin rights to Repository. Could a maintainer with CI-label permission add run-ci (or comment /tag-and-rerun-ci) so the actual Arm64/H100/B200/AOT jobs can run? The H-specific lint and local CUDA validation are already complete.

[liuhao-labs]

Follow-up on the CI gate (2026-10-08): @BBuf @HaiShaw @ispobock @merrymercy could one of you please add the repository run-ci label or comment /tag-and-rerun-ci?

I rechecked the current H head (4c3e4c2350f4d6c4ff2a785e9b294938766a3cc6): the PR is Ready, but still has only sgl-kernel, no formal review, and the check summary is 14 gate/finish failures, 81 skipped jobs, and 5 successes. The gate stops after Missing required label 'run-ci', so no build, AOT, Arm64, H100, or B200 test has started. The author token cannot add the label (403 Must have admin rights to Repository). Once the label is added, I will monitor the real jobs and address any code failures promptly.

[liuhao-labs]

The branch is now refreshed against the current upstream main and pushed as head 5c646b5dc3867bd41c600008d5cca91e5d5085e9 (base 2d26823f9412f3fab4ca15d444118e1c9409da57). The three-way merge was clean and the feature diff remains 15 files (+4127/-81).

@BBuf @HaiShaw @ispobock @merrymercy, the new head's workflows still stop at the same run-ci label gate before any build/test job starts. Could one of you add run-ci or comment /tag-and-rerun-ci for this refreshed head? Once it runs, I will handle any real test failures immediately.

[liuhao-labs]

Final sync note: upstream main moved by four unrelated commits after the previous refresh, so I merged the current base into the branch and pushed head 32475deebb10719d3d8a6dc2ae65ab204b7b5a63 (base 4b384df0c94dbd0fe30627c27c98c0441b4ebd96). The merge is clean; the PR remains exactly 15 files (+4127/-81), with no overlap from those latest main changes.

@BBuf @HaiShaw @ispobock @merrymercy, please add run-ci or comment /tag-and-rerun-ci for this final refreshed head. The current workflows still stop at the label gate before build/test jobs begin.

[liuhao-labs]

Reviewer evidence packet and merge assessment

I added a reproducibility packet with the current source provenance, route logs, test counts, external A/B data, gate log, and explicit limitations:

reviewer evidence packet

My assessment of the evidence:

• Kernel correctness: high confidence. The eight new formats are covered by the focused CUDA matrix and the complete test_gguf.py run; the production smoke records the expected dense M=16/MMQ, dense M=17/dequant, MoE M=127/vector, and MoE M>=128/batched transitions. The old released wheel was also tested with the capability disabled, preserving its fallbacks.
• Routed-MoE performance: medium-to-high confidence. The issue reporter independently reproduced the issue-scale baseline (57.69 vs 56.3 tok/s) and then compared identical H source and serving arguments with only the kernel wheel changed: 60.93 -> 97.37 tok/s prefill (+59.8%), TTFT -40.0%, decode TPOT +0.8%. The route logs show vector versus batched MMQ, so this is not inferred from a capability flag alone.
• Scope of the claim: bounded. The large checkpoint was reconstructed from surviving shards, with two MXFP4 tensors converted to Q8_0; it is not bit-identical. The run is therefore strong routed-MoE evidence, not a full exact-checkpoint or cross-architecture benchmark. 16k was not feasible in the reported unified-memory budget.
• Merge necessity: high for the reported issue. Issue [Perf] No MMQ kernel for I-quant GGUF: 4-6x slower prefill than llama.cpp on the same weights (measured) #35019 remains open; current main still has the I-quant MMQ TODO and no equivalent eight-format MMQ implementation. The capability handshake keeps the pinned released wheel on safe vector/dequant fallbacks, while the new wheel enables the measured MoE path.

The only missing merge evidence is repository-owned: run-ci must be added by a maintainer, then the official wheel/AOT/Arm64/H100/B200 jobs must run and a CODEOWNER must review. The current red checks are still gate failures before any build or test job starts; the latest gate log says PR Draft: false followed by Missing required label 'run-ci'.

[liuhao-labs]

The evidence packet is now aligned with the final refreshed head 0a39ec3a8754e3c4426950a5a93046c048a03f07 and upstream base 6e7beace143964386ce2e4418401b7369f0dabd0. I also updated the gate reference to the latest base job, which shows the same pre-test stop:

PR Draft: false
Require run-ci: true
Missing required label 'run-ci'
Process completed with exit code 1.

The benchmark PNGs and the reviewer matrix are linked in the packet. The packet deliberately separates high-confidence kernel correctness evidence, medium-to-high-confidence routed-MoE A/B evidence, and the still-pending official CI/CODEOWNER review.

[liuhao-labs]

I also uploaded the captured text logs so the route and gate claims are inspectable without relying on screenshots:

• local GGUF route log
• latest PR gate log
• reviewer evidence packet

The route log records capability, dense/MoE dispatch transitions, CUDA Graph replay, old-wheel fallback, and the client-output hash. The gate log records that the latest head is not draft and is stopped specifically by the missing run-ci label.

[liuhao-labs]

Revalidation after main #43236: shared GGUF activation math

I verified a semantic dependency change outside this PR's 15 paths: main
7ff3a112846a3fcc12a27bba09d644885a226e4d enables --use_fast_math on Blackwell
in the shared JIT activation used by fused_moe_gguf. The older AOT-only
tests/first-projection smoke did not cover that policy, so I performed fresh
bounded checks rather than treating the unchanged file list as sufficient.

Pinned report, raw logs, compiler commands, scripts and SHA256 manifest

• Main and main+PR each passed 60 activation comparisons, including every
  finite FP16/BF16 gate value and random inputs at token boundaries, SiLU/GELU,
  FP64 references, and separate empty-cache precise/fast builds. Actual nvcc
  invocations confirm the flag difference and sm_121a target. All meet the
  existing atol=0.01, rtol=0.01 bounds; standalone graph replay is exact.
• Main and main+PR each passed 160 real GGUF two-projection cases: eight IQ
  types, mixed IQ3_S/Q5_0 both directions, FP16/BF16, SiLU/GELU, M=1/127/128/129.
  The production fused_moe_gguf function and default shared activation run
  directly. Fast-vs-precise max output difference is 3.05176e-5. Main+PR vs
  dequant reference max difference is 0.00317383, within original MoE
  atol=1, rtol=0.1; cross-main/PR comparisons also pass 0.01/0.01.
• Current registered CPU routing: 10 passed; activation correctness/out
  subset: 54 passed; unchanged GGUF AOT module: 153 passed. Four full
  main+PR MoE CUDA Graph replays are bit-identical to eager output.

Scope: this run uses Torch 2.11/CUDA13 and freshly compiled GGUF sources in
a separate test namespace, with the auxiliary 0.4.4 wheel recorded by hash.
It is source/component evidence, not current published-wheel/full-wheel ABI
proof. The initial pinned 0.5B server attempt hit the unmodified minimum-version
check (0.4.4 < 0.4.9). I am preparing the official hash-verified 0.4.9 wheel and
matching Torch 2.14.1 in isolation to complete that separate smoke; no guard is
bypassed and no existing service was stopped.

The original PR head remains 0a39ec3a8754e3c4426950a5a93046c048a03f07.
The detached verification merge has tree 36909c6658f0ce1ed346db5b85a3568325b9357d,
still 15 files (+4127/-81) from tested main. Main still uses the IQ vector route
at M>=128; main+PR uses batched MMQ, so #43236 does not replace this fix.
No feature-code change is indicated by these results. The earlier +59.8%
throughput / -40% TTFT remain historical pinned-runtime measurements, not new
latest-main results. Official H100/B200 CI and review remain separate limits.

[liuhao-labs]

Released 0.4.9 ABI and current-source paired serving follow-up

The cached 0.4.4 startup gap from my component report was resolved in an
isolated environment. I downloaded and hash-verified the official aarch64
sglang-kernel 0.4.9 wheel (384a0d2158740f5ab42cfc64c872640bb3c80f3b0ec3b4666bc3efb553254bc4),
matched Torch 2.14.1, and aligned the primary current dependencies including
FlashInfer 0.7.1, CUTLASS DSL 4.8.0 and Triton 3.8.0. The published common_ops
loads successfully; its hash is recorded in both arms.

Pinned runtime report, server/client logs, wheel hashes and scripts

Main 7ff3a112... and its detached merge with unchanged PR H 0a39ec3a...
both loaded the exact pinned Qwen2.5 0.5B IQ3_S file, returned health/generate
HTTP 200, 8 prompt / 16 completion tokens, zero cached tokens, and identical
output token IDs/text. Canonical output hash:
2604448b1bc1b2c9a10961f6b562bed70aab5041db4cb8a7db54e83f9cfde287.
Both final arms used identical arguments and primary packages. No minimum
version guard was disabled or test operator binding substituted for these runs.

Evidence scope: published 0.4.9 does not contain the new capability, so
main+PR reports capability=False and uses normal fallback. This is current
released-wheel ABI + small dense-model serving/compatibility proof, not an
enabled IQ-MMQ full-model result or proof of a newly built full PR wheel's ABI.
The 160x2 real MoE-chain component cases still have the separate fresh-source
test-namespace scope described in my earlier report. An additional published
MoE matrix was prepared, but repeated post-serving GPU utilization was 95-96%
from the existing vLLM service, above the <=10% launch gate, so it was deferred.

The first 0.03 static-memory attempt failed the real KV-budget guard; both
final arms used 0.05 with the same 1024-token cap and 10 GiB container limit.
The raw setup/ABI failures and corrections are retained, including old NCCL,
old optional FlashInfer providers and the client's overly broad startup
traceback check. The passing final logs contain no fatal startup/CUDA/OOM error.
Existing services remained running and their health check returned 200.

The PR branch remains clean at H, with no feature-code change indicated.
The historical +59.8%/-40% numbers remain old pinned-runtime data. Official
full-wheel/H100/B200 CI, enabled large-model testing and formal review remain
separate requirements. The body now distinguishes these evidence scopes and
updates the dependency-pin description. No repeated maintainer ping is made.

[liuhao-labs]

Published-wheel MoE follow-up with matched Torch 2.14.1/CUDA13:

• Latest main 27b68303d043 and a clean main+PR verification merge each passed 32 expert-layer cases using the unmodified official sglang-kernel 0.4.9 production bindings: eight IQ formats, FP16/BF16, SiLU/GELU, M128/E8/top-k2, widths256. All outputs were finite and within the existing atol=1, rtol=0.1 bounds; max reference absolute error was 0.00366211. All 32 paired tensors were bit-identical; peak allocator reserved was 122 MiB per arm.
• Profiler confirmed two real sgl_kernel::ggml_moe_a8_vec calls plus the shared default fast-math activation. Fresh JIT caches, actual --use_fast_math/SM121a compiler commands, wheel/library hashes and full output tensors are recorded. Published capability remains false: this validates expert-layer fallback, not enabled IQ MMQ or a complete served expert model.
• I attempted the complete AOT wheel with matched Torch, one build worker and one NVCC thread. Default 10/14 GiB builds and an all-source/all-default-target SM121a-common_ops build hit actual cgroup OOM in existing FP8 units. The 14 GiB failures were in fp8_gemm_kernel.cu, identical on main and main+PR. No wheel was produced; new full-wheel ABI, capability=true production expert validation and the planned new-wheel 153-test run remain unexecuted. I did not raise memory again on the shared live-service host.
• New main changes have no intersection with the 15 PR paths; GGUF, shared activation/JIT core and AOT build metadata are unchanged by blob. A clean verification merge keeps +4127/-81. Main and the official wheel still lack IQ-MMQ capability. No reproduced GGUF numerical regression or feature-code change was found; original PR head remains 0a39ec3a8754.

Pinned report, raw numeric outputs, exact commands, three Docker OOM events and reproducible scripts

The earlier component/test-namespace results and historical +59.8% prefill / -40% TTFT retain their original scope/runtime; they are not new full-wheel or current-main throughput results. Official H100/B200 CI and the existing run-ci authorization remain separate outstanding requirements. Existing services stayed running and health checks returned 200. PR body now separates the completed evidence from these gaps.