[Fix] Disable trtllm moe backend for draft model for a qucik fix#15002
[Fix] Disable trtllm moe backend for draft model for a qucik fix#15002Fridge003 merged 5 commits intosgl-project:mainfrom
Conversation
samuellees
requested review from
BBuf,
Edwardf0t1,
Fridge003,
HaiShaw,
Ying1123,
ch-wan,
ispobock and
merrymercy
as code owners
Summary of ChangesHello @samuellees, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements a quick fix to resolve an accuracy regression encountered with the DeepSeek V3 draft model. The solution involves temporarily restricting the use of the "flashinfer_trtllm" backend for speculative Mixture-of-Experts (MoE) runners. This change is a stopgap measure, with a deeper investigation into the root cause planned for the future. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a quick fix to disable the flashinfer_trtllm MoE backend for the draft model, addressing an accuracy regression with DeepSeek V3. The change includes an assertion to prevent explicit use of this backend for speculative decoding and modifies the default backend selection logic. My review includes a suggestion to refactor the new logic for improved readability and to avoid a redundant computation.
|
/tag-and-rerun-ci one more time |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
…n_eagle3_npu * 'main' of https://github.com/sgl-project/sglang: (25 commits) [NPU] perf update with kvcache nz & w4a8 quant (sgl-project#14423) [PP Prefill][NIXL] Fix PP mode transfer completion tracking to wait for all ranks (sgl-project#15027) Fix GLM-4.6 tool calls don't support streaming output for arguments i… (sgl-project#13989) feature: adding nightly wheel workflow and indexer (sgl-project#14924) [diffusion] feat: Improve LoRA compatibility by adding unified format detection and diffusers-based normalization (sgl-project#14659) [Fix] Disable trtllm moe backend for draft model for a qucik fix (sgl-project#15002) [diffusion] fix: use NDRotaryEmbedding in flux_2 (sgl-project#15034) Mistral Large 3 NVFP4 support (sgl-project#14485) call check_quantized_moe_compatibility after initialize (sgl-project#13876) Add sgl_router_attempt_http_responses_total for single attempt information (sgl-project#15037) Add error code in prometheus metrics and add X-SMG-Error-Code header (sgl-project#15036) Provide more fine grained error reason for reqwest error (sgl-project#15032) Tiny change http router response format to unify (sgl-project#15031) Tiny unify grpc existing error responses into new format (sgl-project#15030) Add `code` field and unify error responses for router (sgl-project#15028) Super tiny remove unused log_request (sgl-project#15035) Fix decode OOM caused by retraction (sgl-project#14939) [CI]Add gb200 runner back (sgl-project#15024) Add a special label for b200 CI runner that can run kernel tests (sgl-project#15033) Fix regression caused by fa3 block_table (sgl-project#15009) ... # Conflicts: # python/sglang/srt/hardware_backend/npu/attention/ascend_backend.py
4 participants
Motivation
A quick fix for draft model acc regression of DeepSeek V3. I'll have a investigation on this deeper after this PR.
cc @b8zhong
Modifications
Accuracy Tests
python3 -m sglang.launch_server --model-path nvidia/DeepSeek-V3-0324-FP4 --speculative-algorithm EAGLE --tp 4 --quantization modelopt_fp4
+-------------+--------+------------+-----------------+
| Latency (s) | Tokens | Acc Length | Speed (token/s) |
+-------------+--------+------------+-----------------+
| 3.580 | 864 | 2.870 | 241.36 |
+-------------+--------+------------+-----------------+
Benchmarking and Profiling
Checklist