Repository navigation
[Perf] dsv4: Score the V4.1 index logits in bf16 on SM100 - #45880
Merged
Merged
Conversation
hnyls2002
requested review from
BBuf,
DarkSharpness,
HaiShaw,
HydraQYH,
celve and
yuan-luo
as code owners
October 22, 2026 19:34
hnyls2002
changed the title
Collaborator
Author
|
/rerun-test test_dsv41_prefill_sparse_indexer.py test_dense_prefill_indexer.py |
Collaborator
Author
|
/tag-and-rerun-ci |
Contributor
|
🚀 🚀 🚀 🚀 |
…e the sm100 indexer tests bf16-aware
Contributor
|
🚀 🚀 🚀 🚀 |
Contributor
|
🚀 |
zcnrex
added a commit
to zcnrex/sglang
that referenced
this pull request
Oct 23, 2026
Resolve conflicts with bf16 V4.1 index logits (sgl-project#44051, sgl-project#45880), grouped verify rows (sgl-project#45251), the two-level-only prefill graph (sgl-project#45783) and KV-shard index-K writes (sgl-project#44280): the adapter takes a logits_dtype and maps [batch, next_n] paged queries and schedules to grouped varlen rows on SM100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Open
5 tasks
5 of 11 tasks
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fp8_fp4_mqa_logits/fp8_fp4_paged_mqa_logits, so the kernels reduce in bf16. This is the configuration the reference kernels use on SM100: they take bf16 head weights only and emit bf16 logits there. The sparse candidate path already used bf16 weights and logits.index_logits_dtype()/index_weights_dtype()inindex_logits.py), resolved on first use so importing the module does not touch CUDA.SGLANG_OPT_DSV41_BF16_INDEX_LOGITSnow defaults to unset: unset means bf16 on SM100 and fp32 elsewhere; setting it keeps working as an explicit override (true: bf16 logits, with bf16 head weights on SM100 only; false: the fp32 path everywhere). Recipes that set it to 1 are unaffected except that SM100 now also passes bf16 head weights.No change on other architectures (SM90 keeps the torch indexer; SM120 keeps fp32 weights and logits), or anywhere the flag is set to false.
Verification: a dispatch check over SM100 / non-SM100 with the flag unset, true and false gives the expected head-weight and logits dtypes on both the flat and paged calls; on SM100 with no env set the branch resolves to bf16 logits and bf16 head weights at runtime; the existing V4.1 indexer unit tests pass.
flat_index_logits_rows_per_tile), as its docstring intended: with bf16 logits the old formula doubled the rows per tile, so the per-row selection scratch that grows with the tile pushed the scoring peak past the fp32 path's budget. fp32 tiling is unchanged. The SM100 prefill indexer tests now score their reference in the indexer's dtypes and accept block or pick differences only where bf16 scores tie at the selection floor; they pass with bf16 (the SM100 default) and withSGLANG_OPT_DSV41_BF16_INDEX_LOGITS=0.CI States
Latest PR Test (Base): 🚫 Run #39857158297
Latest PR Test (Extra): ❌ Run #39857158423
Latest PR Test (AMD ROCm 10): 🚫 Run #39857158175