Repository navigation
Conversation
With HiCache on, an RL weight sync that runs pause_generation(retract)
and then update_weights_*(flush_cache=True) gets success=False ("Weights
were updated, but the cache was not flushed ..."), and /flush_cache keeps
failing for as long as the engine stays paused.
A write-through still in flight at the pause, or an L3 prefetch started
when the retracted request is requeued, is retired only by
check_hicache_events, which runs from the batch getters and
process_decode_queue. Every paused branch continues before those, so the
entry stays in ongoing_write_through or ongoing_prefetch and
is_fully_idle stays false.
Reap HiCache events in _record_scheduler_state_for_paused_engine, which
all paused branches call, without retrying storage prefetches. A PD
decode worker reaps only with the decode radix cache, as
process_decode_queue does.
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
sohom-cs
requested review from
Ying1123,
hnyls2002 and
xiezhq-hermann
as code owners
November 2, 2026 04:47
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ts-while-paused Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
Main now reads deferred_responses, running_reqs, prefill_launched_reqs and request_receiver on the paused and running paths, and _process_hicache_events reads enable_unified_cache_external_linker, so the bare test schedulers set them. The pause test's scheduler sets the linker flag too, since the paused path now reaps HiCache events. Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With HiCache on (
--enable-hierarchical-cache, defaultwrite_through), an RL weight sync runspause_generation(mode="retract"), thenupdate_weights_from_disk(flush_cache=True). The update returnssuccess=False: "Weights were updated, but the cache was not flushed because requests are still running or queued. Retract or finish them, then call /flush_cache." The requests are already retracted, and/flush_cachefails on every paused iteration, so the only way out iscontinue_generation, which serves the new weights over the old weights' cache.The blocker is not the requests: #40779's paused gate already ignores the requeued ones. It is a HiCache write-through still in
ongoing_write_throughand, with an L3 backend, the prefetch the retract requeue starts for a request with 256 or more uncached tokens. Onlycheck_hicache_eventsretires them, and it runs from the batch getters andprocess_decode_queue, which every paused branch skips. #43758 turned the old assert into this message: the update fails cleanly but cannot succeed while paused.Modifications
All seven paused branches call
_record_scheduler_state_for_paused_engine()on each iteration. It now first calls_process_hicache_events(should_retry_storage_prefetch=False): pending acks are reaped before the idle/active record, and no new L3 attempt starts. PD decode reaps only with the decode radix cache, asprocess_decode_queuedoes. No loop body changes.Considered and rejected: a blocking drain at pause entry, as #30899 and #43282 do. It cannot retire the prefetch the requeue issues after the retract, and it blocks the control path.
Edits the same helper as #46334 and #46307.
Accuracy Tests
New file
test/registered/unit/managers/test_scheduler_paused_hicache_events.py, because #43910/#43911 and #46334 edit the neighbouring test files:event_loop_normal, retract pause, one pending write-through: after one paused iteration the update andflush_cache()succeed;The tree cache, forward,
retract_alland weight load are stubbed. On CPU:test_scheduler_pause_generation.pyandtest_scheduler_hicache_events.pystill passNot run: a GPU job with a real HiCache backend, or several ranks. Every rank receives pause and resume through the same broadcast, so I expect the all-reduce inside
check_hicache_eventsto line up as on an idle running loop.Speed Tests and Profiling
N/A: the same
check_hicache_eventscall an idle iteration makes.Checklist
Developed with AI assistance (Claude); every changed line was reviewed and the tests were run locally by me.
CI States
Latest PR Test (Base): ❌ Run #41538484740
Latest PR Test (Extra): ❌ Run #41538484759
Latest PR Test (AMD ROCm 10): ❌ Run #41538484752
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: