Skip to content

Fix fused DeepSeek top-k capability fallback - #35347

Open
nv-tusharma wants to merge 5 commits into
sgl-project:mainfrom
nv-tusharma:codex/fallback-unsupported-fused-topk
Open

nv-tusharma wants to merge 5 commits into
sgl-project:mainfrom
nv-tusharma:codex/fallback-unsupported-fused-topk

Conversation

@nv-tusharma

@nv-tusharma nv-tusharma commented Aug 18, 2026 •

Copy link
Copy Markdown

Motivation

SGLang currently selects FlashInfer's fused_topk_deepseek based on shape constraints without checking whether the active GPU architecture is supported by that kernel. On A10G (SM86), FlashInfer raises BackendSupportedError during DeepSeek grouped top-k routing instead of allowing SGLang to use its existing fallback.

This was observed in the Dynamo nightly SGLang data-parallel test: https://github.com/ai-dynamo/dynamo/actions/runs/32114902544/job/95672470424

Modifications

  • Read FlashInfer's is_compute_capability_supported metadata before selecting fused_topk_deepseek.
  • Preserve compatibility with FlashInfer versions that do not expose capability metadata.

Accuracy Tests

The numerical implementation is unchanged. Unsupported devices now use the existing SGLang fallback instead of entering a kernel that raises before producing output.

Speed Tests and Profiling

Not applicable. Supported devices retain the fused path; the added check is one device-capability lookup during top-k dispatch.

Validation

  • python3 -m compileall -q python/sglang/srt/layers/moe/topk.py test/registered/kernels/ops/moe/test_fused_topk_deepseek.py
  • GPU pytest not run locally because the available environment does not include PyTorch/CUDA.

Checklist

  • Format code according to the project guidance.
  • Keep the change limited to the capability-aware dispatch guard; no new unit test is needed.
  • No documentation update is needed for this internal dispatch fix.
  • Accuracy and performance impact are described above.
  • Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ❌ Run #32763510057
Latest PR Test (Extra): ❌ Run #32763509769
Latest PR Test (AMD ROCm 7.2): ❌ Run #32763509983

Signed-off-by: Tushar Sharma <trsharm25@gmail.com>
Signed-off-by: Tushar Sharma <trsharm25@gmail.com>
@nv-tusharma
nv-tusharma marked this pull request as ready for review August 18, 2026 15:29
@nv-tusharma

Copy link
Copy Markdown
Author

/tag-and-rerun-ci

Signed-off-by: Tushar Sharma <trsharm25@gmail.com>
@nv-tusharma nv-tusharma reopened this Aug 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants