[Quantization] add humming kernel support for deepseek v4

[jinzhen-lin]

This PR add humming kernels to SGLang. This PR is based on #23754 , adding and improving support for DeepSeek V4 on top of it.

Humming Kenrels: https://github.com/inclusionAI/humming

vLLM supports:

• [Quantization] add humming quantization kernel vllm-project/vllm#34556
• [Quantization] add humming mxfp4 moe backend vllm-project/vllm#41083

Humming is a universal, high-performance quantization kernel (similar to the Marlin kernel), but offers several advantages over Marlin:

• Extensive Quantization Support: Supports all combinations of W{1,2,3,4,5,6,7,8}A{16,8,4} for quantization inference.
• Superior Performance: Humming outperforms Marlin, especially in large batch scenarios and on Hopper GPUs.
• Enhanced JIT Support: Compared to the current Marlin JIT implementation in SGLang, Humming offers faster compilation.
• For DeepSeek V4, Humming supports high-performance W4A8 implementation on Hopper.
• Support DeepEP

Benchmark

Service start command

# marlin w4a16
sglang serve \
  --trust-remote-code \
  --model-path /home/admin/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend marlin \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 12321

# humming w4a16
sglang serve \
  --trust-remote-code \
  --model-path /home/admin/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend humming \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 12321

# humming w4a8
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype": "float8e4m3"}' sglang serve \
  --trust-remote-code \
  --model-path /home/admin/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend humming \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --host 0.0.0.0 \
  --port 12321

Benchmark command

# for prefill
python3 -m sglang.bench_serving \
    --backend sglang \
    --tokenizer /home/admin/DeepSeek-V4-Flash/ \
    --port 12321 \
    --dataset-name random-ids \
    --random-input-len 65536 \
    --random-output-len 1 \
    --num-prompts 128 \
    --max-concurrency 32 \
    --random-range-ratio 1

# for decoding
python3 -m sglang.bench_serving \
    --backend sglang \
    --tokenizer /home/admin/DeepSeek-V4-Flash/ \
    --port 12321 \
    --dataset-name random-ids \
    --random-input-len 1 \
    --random-output-len 4096 \
    --num-prompts 256 \
    --max-concurrency 64 \
    --random-range-ratio 1

Benchamrk result (TPS)

	Prefill	Decoding
Marlin W4A16	7272.49	2286.72
Humming W4A16	9060.34	2278.49
Humming W4A8	9917.43	2335.14

In SGLang, splitkv_mla and paged_mqa are used for the prefill part of DeepSeek V4, and the attention part takes longer than expected. If fixed, Humming is expected to achieve a greater e2e improvement.

[gemini-code-assist]

Code Review

This pull request introduces the "Humming" quantization backend and MoE runner, adding optimized Triton and CUDA kernels for specialized quantization formats like MXFP4. The feedback highlights critical issues such as a potential memory leak in runner registration, possible out-of-bounds memory access in the Triton kernel, and problematic in-place configuration modifications. Additionally, the review suggests fixing a typo in attribute mapping, removing redundant rounding operations, and handling variable data type sizes more accurately during memory allocation.

[fzyzcjy]

cc @Fridge003 for hopper w4a16 kernels

[superuct]

Hello，does DeepSeek-V4 Pro can use the humming kernel?

[jinzhen-lin]

It should be supported, but I haven't actually run it myself yet. Welcome to try and feedback.

[huangzhilin-hzl]

@jinzhen-lin Hi, fix sgl-kernel build error jinzhen-lin#1

[huangzhilin-hzl]

Fix applying the 2604B SwiGLU clamp/checker path jinzhen-lin#2

[huangzhilin-hzl]

fix DeepEP empty-token path error jinzhen-lin#3

[superuct]

It should be supported, but I haven't actually run it myself yet. Welcome to try and feedback.
Yes. I have tried the Pro model. Humming also worked with your latest code.