Minor: style improvement of radix_cache and memory_pool - #395
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
liuxianglong17
pushed a commit
to shun8686/sglang
that referenced
this pull request
Apr 20, 2026
modify testcase
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 2, 2026
…ention-scratch threshold ANALYSE_393 item 1 (concept from ik_llama's -amb). The threshold gates MHA_CHUNKED_KV/MHA_ONE_SHOT (materializes per-token K/V scratch scaling with num_local_heads*(qk_head_dim+v_head_dim)) vs. absorbed MLA. A flat token count is not comparable across ranks under uneven TP (e.g. V4-Flash's [32,16,16] head split) or across models, since the scratch bytes it stands for depend on per-rank head count and head dim. Replaces the flat SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD default path with a per-rank MiB budget (--attn-scratch-budget-mib), converted to a token threshold at attention-layer init using that rank's own geometry (attn_scratch_token_threshold). Default (640 MiB) reproduces the legacy 8192-token default bit-for-bit on the DeepSeek-V3 TP=1 reference geometry. The old env var stays as a deprecated escape hatch: honored verbatim (bypassing the MiB conversion) with a one-line deprecation notice; setting both is a hard error naming both flags (ServerArgs._handle_attn_scratch_ budget_deprecation). Removes the "TODO: Design a finer way to determine the threshold" and replaces it with the invariance constraint it was missing. Tests (test/registered/unit/models/test_attn_scratch_budget.py, 15 cases, all green, CPU-only): default-unchanged pin on the TP=1 reference geometry and an even-TP=4 split (both can-fail: a wrong default-budget constant or flat-8192 semantics on a smaller-head rank breaks the pin), the invariance contract (same MiB budget -> inversely proportional token thresholds across head counts, generalized rather than one hardcoded instance), the deprecated alias honored verbatim plus both-set hard error, and the TODO-replacement comment. ruff and codespell clean on all touched files; existing server_args test suite (test/registered/unit/server_args/, 484 cases) still green.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 2, 2026
…reshold reformulated as a MiB budget (ik -amb concept, ANALYSE_393 adoption 1) — the switch trades absorbed-MLA FLOPs against materialized per-token K/V whose bytes scale with num_local_heads, which differs per rank under uneven TP, so a flat token count meant wildly different bytes per rank; per-rank token threshold = budget / (local_heads*(qk_head_dim+v_head_dim)*2), computed from each rank's own geometry; DEFAULT_ATTN_SCRATCH_BUDGET_MIB=640 reproduces the legacy 8192-token default BIT-FOR-BIT on the file's own DeepSeek-V3 reference geometry (671088640 bytes = exactly 640 MiB); old env honored verbatim as deprecated alias with warning, both-set = hard error naming both; 15 tests incl. red-on-purpose counter-tests and the generalized invariance contract across head counts 1-128; TODO replaced by the invariance constraint (sgl-project#395)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 2, 2026
…fill indexer (sgl-project#449) MiB-budgeted query-row loop over the existing KV-page loop (sgl-project#395 discipline, sgl-project#426 composition preserved); B-fold KV-gather duplication bounded from factor B to the budgeted row count; bit-identical (atol=0) against the unchunked path and the untouched single-pass reference. Collective audit clean under uneven-TP rank divergence. GPU measurement (peak VRAM per rank primary) pending — NOTE_449 §5. Full dedup of the B-fold gather stays open as ANALYSE_447 candidate C.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.