[NPU] Support Qwen3.8-Flash-Next graph mode and MTP

[fishAndShrimp]

Motivation

Enable Qwen3.8-Flash-Next inference on NPU with Qwen Sparse Attention (QSA), graph replay, and NEXTN/MTP speculative decoding. The implementation is now based on mainline, which includes the generic model support merged in #37500.

This PR remains a draft while NPU QSA performance optimization, resolution of conflicts with the latest main, and final validation are in progress.

Modifications

• Add NPU QSA Torch fallbacks for MQA, top-k selection, block expansion, and sparse attention, while retaining shared indexer orchestration.
• Support NPU KV-cache layouts and fixed tensor shapes for graph replay. Keep the live request-to-token mapping in graph metadata and zero masked gathered values before attention.
• Guard CUDA-only hyperconnection dispatch on NPU and disable unsupported hyperconnection compilation.
• Support sigmoid-gated normalization on NPU.
• Make PLE short convolution graph-capturable and handle missing draft token counts in QSA draft extend.
• Skip graph input updates only when capture metadata confirms zero recorded update operators; propagate background update failures to the caller.
• Avoid CPU fallback assertions in the NPU QSA write-planning path.
• Add regression coverage for dispatch, normalization, QSA indexing and padding, PLE convolution, graph replay, and graph update error handling.

Accuracy Tests

Environment

• Hardware: A3, 8 NPU devices; tensor parallel size: 8
• CANN: 9.0.0; torch-npu: 2.10.0
• Model: Qwen3.8-Flash-Next BF16
• Graph replay enabled; NEXTN with 3 speculative steps, top-k 1, and 4 draft tokens
• Evaluated revision: 64b0532544b9f784a5731b1c894d25347acc950b, attributed from the Git history and run timeline; the evaluation output does not embed a Git SHA

Results

Precision	Dataset	Samples	Correct	Accuracy
BF16	GPQA Diamond	198	181	91.41%

Evaluation completed on September 14, 2026, with temperature=0.9, seed=3407, n=1, thinking enabled, and a maximum of 260,000 output tokens. All 198 unique questions have successful evaluation records. The interrupted run was resumed from 197 cached results to complete the remaining question; this is one completed evaluation, not multiple independent runs.

The results above validate the current implementation. Earlier accuracy results from the previous branch are omitted because they were produced with a different upstream baseline and NPU speculative verification behavior.

Server command

Show environment and launch command

The paths below assume the installed NPU toolkit environment. Ports, device offset, and model/dataset paths are placeholders for reproduction. Existing environment variable names, toolkit paths, and backend option values retain their required spelling.

export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset all_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ALL_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32

export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export ASCEND_USE_FIA=1

MODEL_PATH="/PATH/TO/MODEL"

python3 -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --host 127.0.0.1 \
    --port 30000 \
    --tp-size 8 \
    --nnodes 1 \
    --base-gpu-id 0 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 4096 \
    --max-prefill-tokens 52000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 8 \
    --mem-fraction-static 0.85 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --mm-enable-dp-encoder \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4

GPQA Diamond command

Show evaluation command

The following starts a fresh evaluation. The reported run used EvalScope's --use-cache option when resuming its incomplete output directory.

evalscope eval \
  --model "/PATH/TO/MODEL" \
  --api-url http://127.0.0.1:30000/v1 \
  --api-key EMPTY \
  --eval-type openai_api \
  --generation-config '{
        "temperature": 0.9,
        "max_tokens": 260000,
        "seed": 3407,
        "n": 1,
        "timeout": 3600,
        "stream": true,
        "extra_body": {"chat_template_kwargs": {"enable_thinking": true}}}' \
  --datasets gpqa_diamond \
  --dataset-hub Local \
  --dataset-args '{"gpqa_diamond": {"local_path": "/PATH/TO/GPQA", "shuffle": true, "shuffle_choices": true}}' \
  --eval-batch-size 32 \
  --ignore-errors

Speed Tests and Profiling

The QSA implementation currently uses Torch fallbacks. In local TP8 tests with eight concurrent requests and 512 output tokens per request, server steady-state decode throughput was approximately 48–50 tokens/s with non-greedy sampling. This is a workload-specific observation, not a general performance guarantee or the end-to-end throughput of the full GPQA evaluation.

The earlier branch forced NPU speculative target verification to use greedy decoding even for requests with nonzero temperature. The current upstream baseline supports non-greedy MTP sampling (#32495), so the previous branch's higher decode throughput is not a comparison under equivalent sampling semantics. Profiling identified full-vocabulary top-k/top-p probability filtering as a major source of the additional verification cost. Performance work should preserve the requested sampling semantics.

Checklist

• Format the code with pre-commit and complete final style checks.
• Add focused regression tests for the modified behavior.
• Document the evaluation configuration and current accuracy result.
• Resolve conflicts with the latest main and rerun relevant validation.
• Complete PR CI validation.
• Optimize NPU QSA performance and validate accuracy after optimization.
• Provide a standardized performance benchmark.