Skip to content

[Perf] Skip trivial DSV4 nonpaged indexer logits - #33857

Merged
Fridge003 merged 1 commit into
sgl-project:mainfrom
weireweire:perf/dsv4-nonpaged-trivial-rows
Aug 13, 2026
Merged

[Perf] Skip trivial DSV4 nonpaged indexer logits#33857
Fridge003 merged 1 commit into
sgl-project:mainfrom
weireweire:perf/dsv4-nonpaged-trivial-rows

Conversation

@weireweire

@weireweire weireweire commented Aug 6, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Motivation

SGL Top-K v1/v2 directly emits sequential indices when a row has no more candidates than index_topk, so those logits are never read. The DSV4 eager nonpaged path still computed them with DeepGEMM.

Modifications

  • Encode trivial DeepGEMM rows as empty [ks, ks) ranges.
  • Preserve the original lengths used by KV gathering and Top-K.
  • Apply the optimization only to SGL Top-K; other backends keep full ranges.
  • Add one focused <=/> index_topk boundary assertion to the existing plan test.

This targets the current eager NonPagedIndexerPlan; unlike #25400, it does not compact rows or add a host-side decision.

Validation

  • Nonpaged indexer unit test: 7 passed, 12 subtests passed.
  • SGL Top-K v1/v2 GPU checks: mixed and all-trivial cases preserve selected indices.
  • Full pre-commit passed.

Performance

GB300, 8K prefill, index_topk=1024, 80 samples:

  • MQA + Top-K: 0.14361 -> 0.13467 ms (-6.23%)

CI States

Latest PR Test (Base): ❌ Run #31718371894
Latest PR Test (Extra): ❌ Run #31718371557

@weireweire
weireweire force-pushed the perf/dsv4-nonpaged-trivial-rows branch 2 times, most recently from 0a87ef3 to 5f71d1f Compare August 6, 2026 09:56
@b8zhong b8zhong mentioned this pull request Aug 11, 2026
Open
41 tasks
@weireweire
weireweire marked this pull request as ready for review August 12, 2026 03:03
@nvpohanh

nvpohanh commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@Fridge003

Fridge003 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py

@github-actions

github-actions Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py:

⛔ test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py, test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py: Dispatch failed: 422

⛔ test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py: Dispatch failed: 422

@Fridge003

Fridge003 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py

@github-actions

github-actions Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py:

⛔ test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py, test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py: Dispatch failed: 422

⛔ test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py: Dispatch failed: 422

@weireweire
perf(dsv4): skip trivial nonpaged indexer logits
0a6ac06 
SGL Top-K v1/v2 synthesizes sequential indices without reading scores when a row has no more candidates than index_topk. The eager nonpaged prefill path nevertheless asked DeepGEMM to compute logits for every such row.

Represent trivial rows as empty [ks, ks) ranges while preserving the semantic lengths used for KV gathering and Top-K. Keep full ranges for Torch and non-SGL fallbacks.

Validated with the nonpaged indexer unit suite, SGL Top-K v1/v2 mixed and all-empty GPU correctness checks, and an 8K GB300 microbenchmark.
@weireweire
weireweire force-pushed the perf/dsv4-nonpaged-trivial-rows branch from 5f71d1f to 0a6ac06 Compare August 13, 2026 15:59
@Fridge003

Fridge003 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py

@github-actions

github-actions Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run 

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run 

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py

@Fridge003
Fridge003 merged commit 54c44fe into sgl-project:main Aug 13, 2026
123 of 147 checks passed
saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026
@weireweire @saturn-acc
[Perf] Skip trivial DSV4 nonpaged indexer logits (sgl-project#33857)
313f3cc 
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
hanwlax pushed a commit to hanwlax/sglang that referenced this pull request Aug 28, 2026
@weireweire @hanwlax
[Perf] Skip trivial DSV4 nonpaged indexer logits (sgl-project#33857)
948fd2c 
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Fridge003 Fridge003 Fridge003 approved these changes
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner
@Qiaolin-Yu Qiaolin-Yu Awaiting requested review from Qiaolin-Yu Qiaolin-Yu is a code owner
@hebiao064 hebiao064 Awaiting requested review from hebiao064 hebiao064 is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

Assignees

No one assigned

Labels

run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@weireweire @nvpohanh @Fridge003