fix: skip RayVirtualCluster.shutdown when Ray already torn down - #3148
Conversation
01da262 to
76a9892
Compare
ZhiyuLi-Nvidia
added
the
CI:Lfast
|
/ok to test 76a9892 |
yuki-97
left a comment
yuki-97
left a comment
There was a problem hiding this comment.
thanks for root cause and helping fix!
I have some thoughts on whether we can scope down the fix, since other recipes run with run_grpo_nemo_gym.py doesn't meet this error. could you help to take a look?
76a9892 to
5d47e13
Compare
ZhiyuLi-Nvidia
changed the title
|
/ok to test 5d47e13 |
5d47e13 to
5dacef4
Compare
|
/ok to test 5dacef4 |
That's consistent with what we observed. The failure fires only when a Ray-touching del runs after Ray's atexit tears down CoreWorker, and whether that condition is reached seems to depend on cluster count and topology. Colocated GRPO has 1 RayVirtualCluster and stays silent. Non-colocated 2-cluster async GRPO stays silent for the topologies exercised so far. MOPD with teacher_cluster runs with 3 clusters and tips Ray past whatever it silently tolerates, so the CHECK fires. |
MOPD nightly (llm_mopd_qwen3_1_7b_3n8g_megatron_pack) exits 1 after
training completes cleanly:
[C] core_worker_process.cc:88: Check failed: !core_worker_process
Ray's atexit tears down CoreWorker first, then Python's GC runs __del__
on RayVirtualCluster instances, which call remove_placement_group on
the dead runtime — CoreWorker re-init fails the CHECK and the process
aborts.
Fix: return early from shutdown() when ray.is_initialized() is False.
The call is unreachable during normal use (Ray is alive), and the
placement groups are already gone with Ray when this branch fires.
Verified with the mopd-qwen3-1.7b-3n8g-megatron-pack nightly recipe
(job 13917529): exit 0, no core_worker_process.cc:88 CHECK in the
driver log, both metric checks (train/loss, token_mult_prob_error)
PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>
5dacef4 to
c247a8f
Compare
ZhiyuLi-Nvidia
changed the title
|
/ok to test c247a8f |
3 participants
Issue
MOPD nightly (
llm_mopd_qwen3_1_7b_3n8g_megatron_pack) exits 1 after training completes cleanly:Ray's atexit tears down
CoreWorkerfirst, then Python's GC runs__del__onRayVirtualClusterinstances, which callremove_placement_groupon the dead runtime — CoreWorker re-init fails the CHECK and the process aborts.Fix
Return early from
RayVirtualCluster.shutdown()whenray.is_initialized()is False. The guard only fires from__del__during_Py_Finalize; placement groups are already gone with Ray at that point.5 lines, 1 file. No new imports, no semantic change when Ray is alive, no reference retention.
Verified twice end-to-end on the
mopd-qwen3-1.7b-3n8g-megatron-packnightly recipe:🤖 Generated with Claude Code