Log progress while waiting for placement group - #2024
Merged
Merged
Conversation
luccabb
marked this pull request as ready for review
June 5, 2026 20:09
_create_placement_group waited with a bare ray.get(pg.ready()), so when the placement group can't be placed yet (a node's GPUs haven't registered with the GCS, or an autoscaler is still bringing nodes up) training hangs with no output, the last line being 'Creating placement group with N GPUs...'. Poll pg.ready() and log the GPU counts every 30s so the wait is observable. The wait stays unbounded, so behavior on autoscaling clusters is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
luccabb
force-pushed
the
fix-pg-ready-timeout
branch
from
June 5, 2026 20:18
f637230 to
c59a19a
Compare
luccabb
changed the title
luccabb
changed the title
Contributor
Author
|
wondering if its a good idea to timeout? |
zhuzilin
approved these changes
Jun 8, 2026
Contributor
|
thanks a lot for this! yeah the origin logging was not intuitive.
hmm... I've not sure about this... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_create_placement_groupwaits for the placement group with a bareray.get(pg.ready()). When the placement group can't be placed yet — a node's GPUs haven't registered with the Ray GCS, or an autoscaler is still bringing nodes up — this blocks with no output, the last log line beingCreating placement group with N GPUs.... The job looks hung with no indication of whether it's stuck, slow, or scaling. This may underlie some of the "training hangs at startup" reports.Fix
Poll
pg.ready()and log the GPU counts every 30s while waiting:The wait stays unbounded — no timeout, no behavior change — so autoscaling clusters (where a pending placement group is the signal that triggers scale-up, and a cold start can legitimately take many minutes) are unaffected.
🤖 Generated with Claude Code