Repository navigation
[Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction - #41593
Conversation
…he-agnostic policies Under FCFS and the other cache-agnostic policies a waiting request's prefix is never matched, and so never refreshed in the LRU, until the admission loop reaches it. A request queued behind prompts that exhaust the budget therefore ages into the eviction victim, loses its prefix while it waits, and is admitted cold, which lengthens the queue for the requests behind it. Run the existing waiting-queue pre-match for cache-agnostic policies too, bounded by the queue length at which LPM already stops matching, gated by SGLANG_ENABLE_WAITING_PREFIX_REFRESH (default on). Eviction order changes; budget and locks do not. Signed-off-by: Pei Li <peili@nvidia.com>
Signed-off-by: Pei Li <peili@nvidia.com>
|
/tag-and-rerun-ci |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
|
/rerun-failed-ci |
|
Addressed both review comments (a512d92, 40da2dd): the per-round refresh is now a device-tree-only timestamp touch ( On the failing checks: |
The per-round refresh called match_prefix, whose external-cache implementations have side effects per call (FlexKV enqueues a lookup task and can allocate or load KV; hierarchical tiers consult the host). Add BasePrefixCache.refresh_device_prefix, a device-tree-only timestamp touch (no-op by default; RadixCache walks its tree and returns the matched length), and route the cache-agnostic refresh through it. Guard the refresh with is_chunk_cache so --disable-radix-cache deployments do no per-request work. The fast-match path for caches that support it is unchanged. Tests: the refresh never calls match_prefix; a chunk cache triggers neither match_prefix nor refresh_device_prefix. Signed-off-by: Pei Li <peili@nvidia.com>
… into peili/refresh-waiting-prefix-fcfs
UnifiedRadixCache is the default prefix cache, so the refresh must reach it: walk the tree core (which refreshes last_access_time along the path) and apply the walk's bookkeeping actions, skipping the streaming-session shortcut, the component finalizers and the external linker that match_prefix adds. The refresh tests now run against both RadixCache and UnifiedRadixCache. Signed-off-by: Pei Li <peili@nvidia.com>
40da2dd to
5708671
Compare
nvpohanh
left a comment
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] Severity: perf | Confidence: Medium
Issue
The refresh is on by default on the default fcfs policy. Many scheduling rounds with a non-empty queue now walk the tree for up to 128 waiting requests, and each UnifiedRadixCache walk also runs torch.cat on device indices. The Speed Tests section says lpm pays the same cost, but lpm is opt-in, and the PR does not measure this cost. The cache-hit tables also omit the model, GPU, and end-to-end TTFT or throughput.
Fix
On an NVIDIA GPU, compare SGLANG_ENABLE_WAITING_PREFIX_REFRESH=0 and =1 on a colocated fcfs server with about 128 waiting long-prompt requests. Report per-round calc_priority time, TTFT, ITL, and throughput. Name the model and GPU for the existing tables.
[by Claude Code] Severity: style | Confidence: High
Issue
The PR description still describes the first version. It says the refresh reuses match_prefix_for_req, but the head calls the new refresh_device_prefix. It says the disaggregation-mode read is evaluated last, but calc_priority now reads get_disagg() first for every cache-agnostic call. The Modifications list also omits the BasePrefixCache, RadixCache, and UnifiedRadixCache changes and the chunk-cache guard.
Fix
Update Modifications to match the head commit. State whether the cache-hit tables come from the current refresh_device_prefix code or from the earlier match_prefix_for_req version.
|
@peilii could you check if agents' comments make sense? thanks! |
|
/rerun-failed-ci |
Picks up the latest main to refresh CI; the previous PR Test Base run was stuck waiting for a B300 runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] The relevant CI failure (https://github.com/sgl-project/sglang/actions/runs/37420426706/job/112192530393) is not caused by this PR. It is a known breakage on |
Picks up sgl-project#42750, which fixes the unrelated CI failure https://github.com/sgl-project/sglang/actions/runs/37420426706/job/112192530393 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nvpohanh
added
the
bypass-fail-fast
|
/rerun-failed-ci |
|
/rerun-failed-ci |
|
[by Claude Code] All NV pipelines have passed. |
|
@hnyls2002 @xiezhq-hermann could you review this? Thanks! |
main folded match_prefix_for_req into mem_cache/common.py:match_kv_cache. Move _req_radix_key and refresh_waiting_prefix next to it so the matcher and the refresh still build the request key in one place.
|
/rerun-failed-ci |
|
[by Claude Code] All NV pipelines have passed. |
…-only - Refresh after the queue is sorted, head last, so eviction follows the admission order of LOF, random, routing-key and priority scheduling too. - Throttle to once per WAITING_PREFIX_REFRESH_INTERVAL_S (0.5 s); a refresh only has to keep waiting prefixes newer than idle ones. - Skip unless --radix-eviction-policy is lru: other strategies rank by other keys, and MRU would evict the refreshed prefixes first. - refresh_device_prefix returns nothing; its docstring now states the call rate and that host-backed nodes are touched in HiCache mode. - Tests: cover the skips, the throttle and the sorted order; drop the cases the others already cover.
hnyls2002
changed the title
…ments - Restore the fast-match block in calc_priority to main's form. - Rename refresh_device_prefix to refresh_prefix: with a host tier it also touches host-backed nodes. refresh_waiting_prefix takes (req, tree_cache) like match_kv_cache. - Condense the refresh comment; the base docstring no longer names a scheduler constant. - Keep two tests: under LRU the waiting prefixes are evicted in reverse admission order, and under MRU the refresh is skipped.
- refresh_prefix -> touch_prefix (and touch_waiting_prefix, _touch_waiting_prefixes): it only bumps LRU recency, like _touch_node, and does not recompute prefix indices. - The LPM/HRRN fallback and the waiting-prefix touch read one bound, WAITING_QUEUE_PREFIX_MATCH_MAX, instead of two literals of 128.
State where the two bounds come from, keep the touch_prefix contract to two lines, and drop test comments that restate the code.
|
/rerun-test test_radix_cache_hit.py test_hicache_unified_memory.py test_prefill_complete_allocation.py |
|
🚀 🚀 |
main renamed MatchResult.device_indices to device_prefix_len.
… pass Root cause: sgl-project#41593 keeps waiting requests' device prefixes warm by touching them in the radix LRU, but on a PD prefill worker it walks only `waiting_queue`. Requests still in `disagg_prefill_bootstrap_queue`, waiting for decode to pre-allocate their slots, are never touched. When decode is full, the bootstrap queue is the deep one, and those prefixes age out while the requests wait. With DEP2 prefill (9 workers) + DEP16 decode serving an agentic multi-turn trace at 7480 concurrent sessions, the bootstrap queue held ~300 requests per rank for ~100 s. Device-radix hits fell from ~65 % to ~22 % of prompt tokens, and the extra external (Mooncake L3) reloads slowed prefill further, so prefill collapsed after the client's turn burst and did not recover. Fix: `calc_priority` takes the PD prefill bootstrap queue, and on the same refresh pass `_touch_waiting_prefixes` touches up to WAITING_PREFIX_REFRESH_BOOTSTRAP_MAX (512, a module constant next to the existing waiting-queue bounds) bootstrap entries before the waiting queue, in reverse order. SGLANG_ENABLE_WAITING_PREFIX_REFRESH=0 still turns the whole refresh off. The existing gate is unchanged: cache-agnostic policy, LRU eviction, not decode, no fast match, prefix sharing supported. Bootstrap entries are refreshed only on passes that reach `calc_priority`. The caller returns earlier when the waiting queue is empty or no request rows are free, so the bootstrap queue is not refreshed on its own. With more than 128 waiting entries, the entries past the first 128 are still not touched and can end up older than the refreshed bootstrap entries. Validation: GB300, DSv4.1-Flash, 9 DEP2 prefill + DEP16 decode, 7480 concurrent sessions, 1200 s window, SGLang main at 6563312 plus open PRs sgl-project#46497/sgl-project#46593/sgl-project#46592/sgl-project#46404 on both arms: without: 361,815 tok/s/GPU @ p90 interactivity 145.0, TTFT p90 135.9 s; L3 73 % / device 22 % of prompt tokens in minutes 8-20. with: 560,412 tok/s/GPU @ p90 interactivity 161.0, TTFT p90 58.9 s; L3 21 % / device 74 %. test_scheduler_held_row_admission's Scheduler stub gains `disagg_prefill_bootstrap_queue = None` for the new argument.
Motivation
Under the default
fcfspolicy (andlof,random,routing_key) a waiting request is invisible to radix-cache eviction. Its prefix was last matched when its previous turn ran; while it sits in the waiting queue nothing touches that prefix again until the admission loop inget_new_batch_prefillreaches it, and that loop stops at the first request the token budget cannot fit. Eviction is LRU onTreeNode.last_access_time, refreshed only by a prefix match, and only admitted requests holdlock_ref. So the request that is about to be scheduled is, to the LRU, a session idle since its last turn, and it becomes the eviction victim as soon as it has waited longer than the other live sessions have been away.Once that happens the loop closes on itself: the request is admitted cold, needs the whole prompt recomputed instead of the new tokens, holds the batch for several chunks, and every request behind it waits longer, ages past the same threshold, and loses its own prefix. One cold prompt arriving at a rank whose cache is full turns into minutes of recomputation on that rank.
The condition is generic: a cache-agnostic policy, an LRU radix cache whose live working set is at or above the pool, and a waiting-queue tail of seconds. Multi-turn traffic with long cached prefixes is where all three meet.
lpmdoes not show it only because_compute_prefix_matcheshappens to match every waiting request each round while sorting.In internal testing of a disaggregated prefill serving multi-turn traffic with long cached prefixes, with the live working set at or above the radix pool, the cache hit rate at admission was flat for requests admitted quickly and fell with time spent in the waiting queue once a request had waited longer than the idle sessions had been away. Requests in the long-wait tail were admitted mostly cold, and a burst of cold admissions on a rank was usually followed by a storm of low hit rates on that rank. With the first version of this change the hit rate stayed flat across the wait distribution. The head commit is measured end to end below.
Modifications
The refresh only changes eviction order, and only where that order can improve: every other configuration keeps the previous behavior exactly.
schedule_policy.py: after the waiting queue is sorted,calc_prioritycalls_refresh_waiting_prefixes, which touches the cached prefix of the firstWAITING_PREFIX_REFRESH_MAX_QUEUE = 128requests (the depth at whichlpmalready stops matching), tail to head, so the head carries the newest timestamp and eviction follows admission order. Refreshing after the sort makes this hold forlof,random,routing_keyand priority scheduling too. It runs only when all of these hold, and is a no-op otherwise:fcfs,lof,random,routing_key, orlpm/hrrnafter their fallback to FCFS above 128 waiting requests); cache-aware policies already refresh as they match;--radix-eviction-policy lru(the default); other strategies rank by other keys, andmruwould evict the refreshed prefixes first;--disable-radix-cache) and has no fast match path, and the worker is not a decode worker;SGLANG_ENABLE_WAITING_PREFIX_REFRESHis on (default);0restores the previous behavior;WAITING_PREFIX_REFRESH_INTERVAL_S = 0.5s have passed since the last refresh. A refresh only has to keep waiting prefixes newer than idle ones, so it need not run every round.common.py:refresh_waiting_prefix(tree_cache, req)builds the key with_req_radix_key, shared withmatch_kv_cache, so the refresh touches exactly the path admission will match.base_prefix_cache.py:BasePrefixCache.refresh_device_prefix(key) -> None, a no-op by default. It walks the tree and stamps it; with a host tier it also touches host-backed nodes, but it never allocates, loads, or enqueues lookups on a storage backend or external connector, asmatch_prefixmay.radix_cache.py:RadixCache.refresh_device_prefixruns the key preparation ofmatch_prefix, then_match_prefix_helperon the local tree. Subclasses such asFlexKVRadixCacheandPureSWARadixCacheinherit it, so their external lookups are never reached.unified_radix_cache.py:UnifiedRadixCache.refresh_device_prefixrunstree_core.match_prefixplus_apply_cache_actionsfor a possible boundary split, skipping the streaming-session shortcut, the cache-level finalizers and the external linker.environ.py:SGLANG_ENABLE_WAITING_PREFIX_REFRESH = EnvBool(True).test/registered/unit/mem_cache/test_fcfs_waiting_prefix_refresh.py(CPU,base-a-test-cpu), on bothRadixCacheandUnifiedRadixCache: the head's prefix survives when eviction must take a waiting prefix; the head is still refreshed with a queue deeper than the bound; the refresh never callsmatch_prefix; it is skipped with the switch off and undermru,fifoandlfu; it is throttled; it follows the sorted order underlof. AChunkCacheis left alone.test/manual/test_schedule_policy.py: publish a runtime context insetUp, sincecalc_priorityreads the disaggregation namespace.Hit counts (
hit_count, which LFU and SLRU read) are not touched by the refresh; a split during the walk is bookkeeping only.Alternatives considered:
lpmchanges scheduling order and fairness to fix an eviction-order defect;fcfsis the default and stays the default here.lock_refon waiting requests' prefixes would protect them absolutely, but a deep queue could then pin more than the budget and stall admission; refreshing the timestamp keeps every page evictable when nothing else is left.Accuracy Tests
Not applicable: eviction order only, no change to model outputs.
Speed Tests and Profiling
End to end on one H200 per arm,
meta-llama/Llama-3.1-8B-Instruct, default server flags plus--max-total-tokenswhere noted to put the radix cache under pressure. The base is the merge base with main (54b551d); both arms ran at the same time with the same client seed. Multi-turn traffic isbenchmark/hicache/bench_multiturn.py --num-clients 128 --max-parallel 128 --request-length 1024 --output-length 128 --num-rounds 8 --request-rate 64 --ready-queue-policy fifo --disable-random-sample --disable-auto-run, so every pending turn waits in the server queue.No-reuse overload (
bench_serving --dataset-name random-ids --random-input-len 4096 --random-output-len 256 --num-prompts 1200 --request-rate 12, default pool): throughput 6.59 / 6.61 req/s and median ITL 35.48 / 35.49 ms (main / PR).CPU cost of one refresh with 128 waiting requests, measured on CPU: 0.9 ms (2k-token prompts) to 7.2 ms (32k-token prompts) on
RadixCache, 1.6 to 6.8 ms onUnifiedRadixCache; with the 0.5 s throttle that is at most about 14 ms per second, and nothing when the queue is empty. The Rust tree core was not exercised.Checklist
CI States
Latest PR Test (Base): ⏳ Run #37861667395
Latest PR Test (Extra): ❌ Run #37861667013
Latest PR Test (AMD ROCm 10): ⏳ Run #37861667407