[Feature] Enable E8M0 by Default on Hopper for DeepGEMM, 5% E2E throughput improvement

[yewentao256]

Purpose

1. Reduce the complexity
2. Speed up!

Test

Unit Test

~/vllm-source/tests/kernels/moe$ pytest test_deepgemm.py

collected 12 items                                                                                                                     

test_deepgemm.py ............                                                                                   [100%]

============================================ 12 passed in 96.91s (0:01:36) ============================================

collected 32 items                                                                                                    

test_batched_deepgemm.py ................................                                                       [100%]

============================================ 32 passed in 79.44s (0:01:19) ============================================

Acc

lm_eval --model vllm --model_args "pretrained=Qwen/Qwen3-30B-A3B-FP8,max_model_len=32768,enforce_eager=True" --trust_remote_code --tasks gsm8k --num_fewshot 5 --batch_size auto

# now
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8560|±  |0.0097|
|     |       |strict-match    |     5|exact_match|↑  |0.8939|±  |0.0085|

# main
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8097|±  |0.0108|
|     |       |strict-match    |     5|exact_match|↑  |0.8870|±  |0.0087|

Perf

vllm bench throughput --model Qwen/Qwen3-30B-A3B-FP8 --load-format dummy --input-len 1000 --output-len 100 --trust_remote_code --enable-expert-parallel

# now
Throughput: 35.84 requests/s, 39427.04 total tokens/s, 3584.28 output tokens/s
# main
Throughput: 33.89 requests/s, 37275.64 total tokens/s, 3388.69 output tokens/s

[gemini-code-assist]

Code Review

This pull request enables E8M0 by default on Hopper for DeepGEMM by unifying the environment variables for Hopper and Blackwell GPUs. The changes correctly remove the Hopper-specific environment variable VLLM_USE_DEEP_GEMM_E8M0_HOPPER and update the logic to use the generic VLLM_USE_DEEP_GEMM_E8M0 for both. My review includes suggestions to improve code maintainability by updating an outdated comment and removing a redundant conditional check.

[djmmoss]

How does this work on Hopper?

If I'm not mistaken: https://github.com/deepseek-ai/DeepGEMM/blob/239112cb4cd4e52587c662624aee6beda8bd9518/csrc/apis/layout.hpp#L22 and https://github.com/deepseek-ai/DeepGEMM/blob/239112cb4cd4e52587c662624aee6beda8bd9518/csrc/apis/layout.hpp#L32 disable the ue8m0 layout on hopper regardless of the flag.

[benchislett]

What does this flag actually do on hopper? Looking through the DeepGEMM code at a glance, it seems like E8M0/disabled doesn't change any behaviour. Could you help me understand what this flag controls and how it leads to the speedup you measured?

[yewentao256]

How does this work on Hopper?

https://github.com/deepseek-ai/DeepGEMM/blob/239112cb4cd4e52587c662624aee6beda8bd9518/csrc/jit_kernels/impls/smxx_layout.hpp#L113

I think return get_mn_major_tma_aligned_tensor(sf); doesn't change the e8m0, it just make the TMA-aligned and MN-major tensor?

[yewentao256]

What does this flag actually do on hopper? Looking through the DeepGEMM code at a glance, it seems like E8M0/disabled doesn't change any behaviour. Could you help me understand what this flag controls and how it leads to the speedup you measured?

E8M0 is false by default, which means if the model doesn't have scale_fmt: ue8m0 in the config, we will not convert to e8m0 result, which would be slower.

The conversion could be found

vllm/vllm/model_executor/layers/quantization/utils/fp8_utils.py

Line 813 in 5c057e0

def requant_weight_ue8m0_inplace(

[benchislett]

makes sense, thanks for the insight!

[yewentao256]

@youkaichao CC

[bnellnm]

LGTM

[youkaichao]

to clarify, there's activation e8m0 and weight e8m0, two separate things.

on hopper, we should only use activation e8m0 when the model config says scale_fmt: ue8m0 , which is only for deepseek v3.1 or later.

[mgoin]

@youkaichao why did you merge this then? @yewentao256 showed improvements for Qwen/Qwen3-30B-A3B-FP8 which doesn't have scale_fmt: ue8m0, so my understanding is this PR is applying activation e8m0 for any model using DeepGEMM on Hopper