Skip to content

[sglang-miles] Fix post_process_weights deadlock in RL in-place mode - #29670

Merged
ByronHsu merged 1 commit into
sgl-project:sglang-miles-v0.5.13from
ByronHsu:byron/fix-v0513-pause-aware-post-process
Jun 29, 2026
Merged

[sglang-miles] Fix post_process_weights deadlock in RL in-place mode#29670
ByronHsu merged 1 commit into
sgl-project:sglang-miles-v0.5.13from
ByronHsu:byron/fix-v0513-pause-aware-post-process

Conversation

@ByronHsu

@ByronHsu ByronHsu commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Restore the pause-aware locking behavior for post_process_weights on the sglang-miles-v0.5.13 branch.

This is the post_process_weights hunk from the earlier pause-aware deadlock fix in #22754, which appears to have been lost during the v0.5.13 tokenizer_control_mixin.py refactor.

sglang-miles-v0.5.12 skipped model_update_lock.writer_lock when the engine was already paused. During the v0.5.13 refactor to tokenizer_control_mixin.py, post_process_weights became an unconditional writer-lock acquisition. That can deadlock with pause_generation(mode="in_place"):

active /generate holds model_update_lock.reader_lock
pause_generation(in_place) preserves the request and pauses progress
post_process_weights waits for writer_lock
continue_generation is only called after post_process_weights

The result is that health generation stops making progress and post-load processing can time out.

Fix

  • Check self.is_pause under is_pause_cond.
  • Use model_update_lock.writer_lock only when the engine is not paused.
  • Use nullcontext() while paused, matching the pause-aware behavior from the earlier miles branch.

Validation

python3 -m py_compile python/sglang/srt/managers/tokenizer_control_mixin.py

CI States

Latest PR Test (Base): ❌ Run #28393955205
Latest PR Test (Extra): ❌ Run #28393954957

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ByronHsu
ByronHsu force-pushed the byron/fix-v0513-pause-aware-post-process branch from a323af7 to a705d50 Compare June 29, 2026 18:19
@ByronHsu ByronHsu changed the title [sglang-miles] Fix pause-aware post_process_weights locking [sglang-miles] Fix post_process_weights deadlock in RL in in-place mode Jun 29, 2026
@ByronHsu ByronHsu changed the title [sglang-miles] Fix post_process_weights deadlock in RL in in-place mode [sglang-miles] Fix post_process_weights deadlock in RL in-place mode Jun 29, 2026
@ByronHsu
ByronHsu force-pushed the byron/fix-v0513-pause-aware-post-process branch from a705d50 to 23d9898 Compare June 29, 2026 18:28
@ByronHsu
ByronHsu merged commit d15e190 into sgl-project:sglang-miles-v0.5.13 Jun 29, 2026
38 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants