Enable parts of CI using github labels & allow workflow dispatch with infinite num epochs - #7
Merged
Conversation
fzyzcjy
marked this pull request as ready for review
October 10, 2025 10:20
fzyzcjy
changed the title
fzyzcjy
added a commit
that referenced
this pull request
Mar 1, 2026
- #1: Move argparse import to top-level in parallel_utils.py - #2: resolve_megatron_path returns Path instead of str - #3: Log non-zero comparator exit code in compare.py - #4: Use `is not None` instead of truthiness for consistency in compare.py - #5: Replace dataclasses.asdict with explicit kwargs, remove type: ignore - #6: Add PreTrainedTokenizerBase type annotation in prompt_utils.py - #7: Replace assert with ValueError in _resolve_raw_text - #8: Handle default_factory in DataclassArgparseBridge - #9: Document bool | None as unsupported in argparse_utils docstring - #10-#12: Add missing type annotations (model_provider, forward_backward_func, losses, records)
GuanxingLu
pushed a commit
to GuanxingLu/miles
that referenced
this pull request
Apr 21, 2026
… infinite num epochs (radixark#7)
yueming-yuan
added a commit
that referenced
this pull request
Aug 22, 2026
Its first CI run ever -- it had been disabled since it was added -- hung in tools/convert_hf_to_torch_dist.py, not in training. Four ranks reached the barrier in Megatron's dist_checkpointing execute_sync and rank 0's NCCL watchdog killed the job after the 600s default: WorkNCCL(SeqNum=9, OpType=ALLREDUCE, NumelIn=1) timed out #0 barrier torch/distributed/distributed_c10d.py:5030 #2 execute_sync Megatron-LM/.../strategies/async_utils.py:99 #7 main tools/convert_hf_to_torch_dist.py:131 Not a Blackwell problem: test_glm5_2_744b_a40b_5layer_nvfp4 passed in the same job, and nvfp4 kernels ran fine. It converts with --nproc-per-node 1, so it has no cross-rank barrier to time out on; this one uses 4. Not a slow disk either. The save wrote 22G of a ~50B-parameter MoE in 13 minutes, about 28 MB/s, while the host's NVMe measures 2.8 GB/s write and 3.2 GB/s read. Whatever stalls that path, raising the NCCL timeout only makes it hang longer. scripts/run_joy_ai_llm_flash.py stays -- it is a documented launch script and this only removes the CI wrapper that never worked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.