Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
22b209a
Support unified memory with speculative decoding
Aug 26, 2026
e887a18
fix: use hybrid SWA draft flag for pool setup
merrymercy Sep 1, 2026
ad6c3b4
Use shared byte capacity for unified SWA pools
Aug 27, 2026
1aac878
Avoid double-rounding unified SWA demand
ZYHowell Aug 27, 2026
7ee2c74
Reject unsupported unified pool scheduler modes
Aug 27, 2026
c54b97d
Remove unified pool gate-only tests
Aug 28, 2026
8892c52
Limit shared-capacity changes to enabled paths
Aug 28, 2026
48b2a24
Fix unified SWA eviction over-reclaim
Aug 28, 2026
139a9a1
Remove added unified eviction planner test
YhZhuang Aug 28, 2026
9c31d3d
Remove low-signal unified capacity tests
Sep 1, 2026
1599357
Adapt shared capacity to current main
Sep 1, 2026
c2296be
Address unified SWA admission feedback
Sep 1, 2026
05cfb81
Support unified memory page envelopes in PD transfers
Aug 27, 2026
c4551a5
Use decode request pool for unified SWA PD
Aug 27, 2026
623f762
Reject unified Mooncake speculative transfers
Aug 27, 2026
c9b4c29
Simplify unified draft index plumbing
Aug 28, 2026
5a7d1ef
Fix unified SWA PD capacity accounting
Sep 1, 2026
08a06eb
Prefer same-side eviction for unified SWA
ZYHowell Sep 2, 2026
6c46fed
Merge updated shared-capacity branch into PD page envelopes
ZYHowell Sep 2, 2026
fe5787a
Prefer SWA tombstones before unified FULL eviction
Sep 2, 2026
5d19c4f
Merge updated shared-capacity policy
Sep 2, 2026
e6e491b
Restore upstream unified cache eviction policy
Sep 2, 2026
cc12f23
Merge restored eviction policy into PD page envelopes
Sep 2, 2026
66fed93
Use joint capacity accounting for unified SWA
Sep 2, 2026
aa6d1cd
Merge joint capacity accounting into PD page envelopes
Sep 2, 2026
c41b068
Add unified capacity regression coverage
Sep 4, 2026
43ecd06
Use physical SWA indices for PD transfer
Sep 4, 2026
6fe1e7d
Merge unified capacity fixes into PD envelopes
Sep 4, 2026
d315b7d
Merge remote-tracking branch 'origin/main' into yonghao/ump-shared-ca…
Sep 4, 2026
7fadf8e
Merge commit 'd315b7d65e' into yonghao/ump-pd-page-envelopes
Sep 4, 2026
940d41b
Update PD priority test fixture for draft indices
Sep 4, 2026
d01513d
Update HiSparse test fixture for unified memory
Sep 4, 2026
25d1e3b
Merge commit 'd01513db8e' into yonghao/ump-pd-page-envelopes
Sep 4, 2026
8bc1586
Plan unified FULL/SWA reclaim against physical capacity
Sep 8, 2026
7f10ff8
Merge shared-capacity fixes into page-envelope transfers
Sep 8, 2026
653c010
Restore unified pool cleanup and scoped compatibility gates
Sep 8, 2026
07df3f4
Merge shared-capacity cleanup into PD page envelopes
Sep 8, 2026
f83adaf
Merge upstream main and adapt unified capacity to allocator refactors
Sep 10, 2026
b32e13e
Merge shared capacity refresh and adapt PD envelope allocators
Sep 10, 2026
f604efb
Keep HiSparse fixture compatible with Python 3.10
Sep 10, 2026
e09999e
Merge Python 3.10 fixture compatibility fix from shared capacity
Sep 10, 2026
70312b5
Remove unrelated UMP diff churn
Sep 10, 2026
054c220
Merge shared-capacity cleanup into PD page envelopes
Sep 10, 2026
4e0e1c3
Fix unified pool admission rounding
Sep 12, 2026
69747f6
Fix full-envelope PD transfers and merge capacity fixes
Sep 12, 2026
70671d5
Refactor unified SWA budget handling
Sep 13, 2026
66a2172
Merge shared-capacity cleanup
Sep 13, 2026
cb88f1f
Centralize decode SWA reservation routing
Sep 13, 2026
29545f8
Simplify shared SWA budget paths
Sep 13, 2026
8d4460c
Merge shared SWA budget cleanup
Sep 13, 2026
4d1fa0f
Fix unified capturer indexing and asymmetric SWA reservation
Sep 13, 2026
d0a10e5
Merge shared-capacity fixes and repair unified CPU copy API
Sep 13, 2026
567f4f5
Consolidate unified allocation, admission and capacity policies
Sep 13, 2026
6066493
Merge capacity cleanup and cover real SWA tail allocation
Sep 13, 2026
6c7be62
Refactor SWA admission into allocator-owned memory budgets
ch-wan Sep 13, 2026
4fdb2a7
Fix unaligned unified SWA prefill admission
Sep 14, 2026
5a8f410
Merge unified prefill rounding fix into PD support
Sep 14, 2026
089673a
Refactor unified SWA allocators into sibling capacity policies
Sep 14, 2026
121fdde
Clean up unified PD rejection and transfer interfaces
Sep 14, 2026
3c9040a
Merge reviewed shared-capacity cleanup into PD layer
Sep 14, 2026
564845d
Use shared byte capacity for unified SWA pools
Sep 14, 2026
a7084b3
Keep HiSparse fixture compatible with Python 3.10
Sep 10, 2026
b505c11
Remove unrelated UMP diff churn
Sep 10, 2026
99ee780
Fix unified pool admission rounding
Sep 12, 2026
8fd4aa4
Refactor unified SWA budget handling
Sep 13, 2026
fa6c95a
Simplify shared SWA budget paths
Sep 13, 2026
d9c7447
Fix unified capturer indexing and asymmetric SWA reservation
Sep 13, 2026
6cf450c
Consolidate unified allocation, admission and capacity policies
Sep 13, 2026
0a80658
Refactor SWA admission into allocator-owned memory budgets
ch-wan Sep 13, 2026
1f3b496
Fix unaligned unified SWA prefill admission
Sep 14, 2026
f208606
Refactor unified SWA allocators into sibling capacity policies
Sep 14, 2026
f0a0085
Fix shared prefill deferral and capture and draft capacity bounds
ch-wan Sep 14, 2026
ab2ea29
Validate inherited unified allocator PD and free-group contracts
ch-wan Sep 14, 2026
e3fa99e
Merge updated shared-capacity branch into UMP PD support
Sep 14, 2026
ab7d973
Explain SWA prefill reservation and chunking safety
Sep 14, 2026
ea634ec
Merge shared-capacity updates and simplify PD reservation checks
Sep 14, 2026
81d00a3
Refresh shared KV budgets during decode admission
Sep 14, 2026
405e92b
Port profiled SWA byte-budget fix from #36729
Sep 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions python/sglang/srt/arg_groups/fields/memory.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,8 @@ class Memory(msgspec.Struct):
"Replace the statically-partitioned hybrid-model pools (full-attn KV + "
"SWA/Mamba state) with one byte buffer split dynamically between "
"sub-pools. Requires the Triton attention / linear-attn / Mamba "
"backends. PD disaggregation is supported over mooncake at equal "
"attention TP with pp=1; not yet compatible with hierarchical / "
"host-tiered KV cache, prefill cuda-graph capture, or speculative "
"decoding other than DSPARK.",
"backends. Supported PD-disaggregation and speculative-decoding "
"configurations are validated at startup.",
] = False
enable_session_radix_cache: A[
bool,
Expand Down
27 changes: 22 additions & 5 deletions python/sglang/srt/arg_groups/kv_cache_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -200,18 +200,26 @@ def handle_unified_memory_pool(server_args: Any) -> None:
if not cfg.enable_unified_memory:
return
if cfg.disaggregation_mode != "null":
# Constraints of the whole-envelope transfer; see
# UnifiedMLATokenToKVPool.get_contiguous_buf_infos.
assert cfg.disaggregation_transfer_backend == "mooncake", (
"--enable-unified-memory with PD disaggregation supports only "
"the mooncake transfer backend; got "
# Constraints of the whole-envelope transfer; see the unified MHA and
# MLA pool get_contiguous_buf_infos implementations.
supported_backends = server_args._unified_memory_pd_transfer_backends()
assert cfg.disaggregation_transfer_backend in supported_backends, (
"--enable-unified-memory with PD disaggregation supports only these "
f"transfer backends: {', '.join(sorted(supported_backends))}; got "
f"{cfg.disaggregation_transfer_backend!r}."
)
assert cfg.pp_size == 1, (
"--enable-unified-memory with PD disaggregation does not support "
"pipeline parallelism (whole-envelope transfer has no per-layer "
"entries to subset)."
)
assert not (
cfg.disaggregation_transfer_backend == "mooncake"
and cfg.speculative_algorithm is not None
), (
"--enable-unified-memory with PD disaggregation does not support "
"speculative decoding with the Mooncake transfer backend."
)
assert not envs.SGLANG_DISABLE_LAZY_COMPACTION.get(), (
"--enable-unified-memory with PD disaggregation requires lazy "
"compaction; unset SGLANG_DISABLE_LAZY_COMPACTION."
Expand All @@ -222,6 +230,15 @@ def handle_unified_memory_pool(server_args: Any) -> None:
"ships host/C4 rows straight from the allocator, bypassing the "
"virtual->physical translation the unified pool needs."
)
assert cfg.disaggregation_decode_retraction_backup != "host_pool", (
"--enable-unified-memory with PD disaggregation does not support "
"--disaggregation-decode-retraction-backup=host_pool; use "
"cpu_tensor (the automatic default for unified pools)."
)
assert not cfg.disaggregation_decode_enable_offload_kvcache, (
"--enable-unified-memory with PD disaggregation does not yet support "
"--disaggregation-decode-enable-offload-kvcache."
)
assert cfg.speculative_algorithm in (None, "DSPARK"), (
"--enable-unified-memory only supports --speculative-algorithm "
"DSPARK (chain draft); other speculative algorithms are not yet "
Expand Down
3 changes: 3 additions & 0 deletions python/sglang/srt/arg_groups/overrides.py
Original file line number Diff line number Diff line change
Expand Up @@ -1808,6 +1808,9 @@ def post_capture_kv_sizing_planned(server_args: Any) -> bool:
mla_enabled = use_mla_backend(server_args)
if not envs.SGLANG_ENABLE_POST_CAPTURE_KV_SIZING.get():
return False
# Unified arenas are fully backed before capture and cannot resize afterward.
if cfg.enable_unified_memory:
return False
if cfg.device != "cuda":
return False
if cfg.dcp_size != 1:
Expand Down
Loading
Loading