[NPU] Enable MXFP8 low-latency DeepEP dispatch for Fp8MoEMethod FP4 experts

[iridiumine]

Motivation

On NPU, FP4-expert models routed through Fp8MoEMethod ( is_fp4_expert=True , e.g. MiMo-V2.5-Pro with modelopt_fp4) delegate execution to NPUW4A8MXFP4MoEMethod kernels via the ASCEND MoE runner. However, process_weights_after_loading only runs Fp8MoEMethod 's own body, so the DeepEP dispatcher quant config was never set — dispatch fell back to BF16 and a separate DynamicMxQuant kernel ran before every grouped matmul (~15ms/layer at 16K-token prefill).

The native FP4 path ( NPUW4A8MXFP4FusedMoEMethod , DSV4) already sets the same dispatcher config; the Fp8MoEMethod path was simply missing it.

Modifications

• Declare the dispatch wire format as a DISPATCHER_QUANT_CONFIG class attribute on NPUW4A8MXFP4MoEMethod ( low_latency=mxfp8 ; normal stays BF16 since A5 MXFP8 normal dispatch is intranode-only). The kernel that consumes the activations now owns the declaration, instead of each caller hand-writing platform-specific if/else patches.
• In Fp8MoEMethod.process_weights_after_loading , pick the config up from the attached layer.w13_kernel when it declares one; otherwise keep the original checkpoint-dtype config. No _is_npu / is_fp4_expert branching — the attached kernel already encodes both.
• Reuse the same declaration in NPUW4A8MXFP4FusedMoEMethod , replacing its hand-written copy.
  With this, A5 low-latency dispatch quantizes activations to MXFP8 in-kernel (with E8M0 block scales) and the GMMs consume them directly; the three previously duplicated config dicts collapse into a single kernel-owned declaration.

Accuracy Tests

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export HCCL_BUFFSIZE=300
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export ASCEND_USE_FIA=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export DEEPEP_HCCL_BUFFSIZE=2500
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export HCCL_HOST_SOCKET_PORT_RANGE=auto
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1

sglang serve \
        --model-path $MODEL_PATH \
        --served-model-name $MODEL_PATH \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --mem-fraction-static 0.92 \
        --tp-size 8 --nnodes 1 --node-rank 0 \
        --chunked-prefill-size 8192 --max-total-tokens 600000 \
        --max-running-requests 32 \
        --moe-a2a-backend deepep --deepep-mode auto \
        --cuda-graph-bs-decode 1 2 4 8 16 \
        --speculative-algorithm DFLASH \
        --speculative-draft-model-path $DFLASH_PATH \
        --speculative-num-draft-tokens 8 \
        --dp-size 2 --enable-dp-attention --enable-dp-lm-head \

 DeprecationWarning: sglang.test.few_shot_gsm8k is deprecated. Use sglang.test.run_eval with eval_name='gsm8k' instead.
  metrics = run_eval(args)
100%|█████████████████████████████████████████████████████████████████████████████████| 200/200 [00:37<00:00,  5.28it/s]
Accuracy: 0.970
Invalid: 0.010
Latency: 38.289 s
Output throughput: 604.323 token/s
metrics={'accuracy': np.float64(0.97), 'invalid': np.float64(0.01), 'latency': 38.289156210026704, 'output_throughput': 604.3225364663596}
metrics['accuracy']=np.float64(0.97)

Speed Tests and Profiling

MiMo-V2.5-Pro-FP4 (DFlash), single-node A5, DeepEP low-latency, prefill with 16K-token random inputs.

Per-layer kernel profile (avg ms, before → after):

Kernel	Before	After
DynamicMxQuant (post-dispatch)	15.14	eliminated
MoeLowLatencyDispatchV2	33.16	30.63
GroupedMatmulSwigluQuantV2	30.37	27.60
Layer total	238.6	222.7 (-6.7%)

End-to-end (bench_serving, 128 requests, rate=1.0, saturated queueing):

Metric	Before	After
Mean TTFT (ms)	11633	9095 (-21.8%)
P99 TTFT (ms)	18856	16405

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #37717189731
Latest PR Test (Extra): ❌ Run #37717189408
Latest PR Test (AMD ROCm 10): ❌ Run #37717189609

[sglang-npu-bot]

/tag-and-rerun-ci

[iridiumine]

/rerun-failed-ci

[iridiumine]

/rerun-failed-ci

[iridiumine]

/rerun-failed-ci

[iridiumine]

/rerun-failed-ci

[iridiumine]

/rerun-failed-ci

[OrangeRedeng]

Hi! My PR include #26408 an alternative solution that makes it possible to automatically determine the type of output dispatcher (fixing a long‑broken strategy), and it is implemented for both normal and low‑latency modes.

[OrangeRedeng]

I’d also prefer a single interface for interacting with the dispatcher type, for example the change introduced in PR #39589 has now broken automatic detection of the dispatcher output dtype in the TP (tensor parallelism) case.