Add end-to-end accuracy tests to CI - #2
Merged
Merged
Conversation
fzyzcjy
force-pushed
the
feat/acc_test_ci
branch
from
October 10, 2025 00:36
6b6c2e8 to
e7bcb25
Compare
fzyzcjy
added a commit
that referenced
this pull request
Oct 10, 2025
gongyisheng
referenced
this pull request
in gongyisheng/miles
Feb 20, 2026
[chore] fix offload train failure
fzyzcjy
added a commit
that referenced
this pull request
Mar 1, 2026
- #1: Move argparse import to top-level in parallel_utils.py - #2: resolve_megatron_path returns Path instead of str - #3: Log non-zero comparator exit code in compare.py - #4: Use `is not None` instead of truthiness for consistency in compare.py - #5: Replace dataclasses.asdict with explicit kwargs, remove type: ignore - #6: Add PreTrainedTokenizerBase type annotation in prompt_utils.py - #7: Replace assert with ValueError in _resolve_raw_text - #8: Handle default_factory in DataclassArgparseBridge - #9: Document bool | None as unsupported in argparse_utils docstring - #10-#12: Add missing type annotations (model_provider, forward_backward_func, losses, records)
fzyzcjy
added a commit
that referenced
this pull request
Mar 1, 2026
- Build next-token labels using global_input_ids gather with position_ids for correct CP>1 zigzag-sliced label construction (Bug #2) - Construct explicit causal lower-triangular attention_mask instead of passing None (Bug #4) - Use ignore_index=-100 in loss_func for CP-masked positions - Return global_input_ids in batch dict for cross-config reference
fzyzcjy
pushed a commit
that referenced
this pull request
Mar 19, 2026
[rollout] feat: implement partial rollout feature on rollout engine side
fzyzcjy
added a commit
that referenced
this pull request
Mar 19, 2026
GuanxingLu
pushed a commit
to GuanxingLu/miles
that referenced
this pull request
Apr 21, 2026
5 tasks
guapisolo
added a commit
that referenced
this pull request
Jun 6, 2026
Dockerfile_GB300 is unreferenced (build.py / justfile / workflows build the main docker/Dockerfile with build-args). GB300 / arm64 cu130 is documented in the main Dockerfile header (variant #2). Its Blackwell sm_103a-specific steps (patch_fla_blackwell, Triton-from-source, explicit sgl-kernel cu130, TE core_cu13 from source) are intentionally dropped per maintainer decision. Also remove the now-orphaned patches only GB300 consumed: - docker/patch/cu13/patch_fla_blackwell.py (GB300-only fla/Triton fix) - docker/patch/latest/{megatron,sglang}.patch (GB300 PATCH_VERSION=latest) Kept: docker/patch/cu13/te_fa2_sm103_whitelist.patch — the main Dockerfile still applies cu13/*.patch under ENABLE_CUDA_13=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fzyzcjy
added a commit
that referenced
this pull request
Jun 16, 2026
…ght recorder HACK ft-hang-repro (revert manually): deterministically reproduce the bug #2 wedge where a cell dying inside update_weights leaves a survivor cell hung in its own unguarded update_weights collective. - hang_repro_hack.maybe_kill_at_update_weights: env-gated (MILES_FT_HACK_KILL_AT_UPDATE_WEIGHTS=N) self-segfault on the Nth update_weights call on rank 0 of the active source cell. - mixin.update_weights: call it after engine-pause, before the first all_gather (matches r12's observed death point). - execution.py: enable torch NCCL flight recorder (trace buffer + dump-on-timeout + on-demand pipe dump + desync debug) and forward the kill env to actors. - scenario_ft_random: when the deterministic kill is armed, skip the random injector and its post-run assertion so the wedge reproduces with no injection noise.
fzyzcjy
added a commit
that referenced
this pull request
Jun 16, 2026
…ecorder plumbing The deterministic update_weights-kill harness (hang_repro_hack) and the NCCL flight-recorder / kill-env wiring were only needed to root-cause bug #2; that fix is in and validated across the e2e suite, so drop the throwaway debug code: - delete miles/utils/test_utils/hang_repro_hack.py - restore mixin.py / broadcast.py (drop the maybe_kill_at_update_weights call sites) - restore execution.py (drop the flight-recorder env block and kill-env forwarding) - restore scenario_ft_random.py (drop the injector-skip; the random injector always runs)
yueming-yuan
added a commit
that referenced
this pull request
Aug 22, 2026
Its first CI run ever -- it had been disabled since it was added -- hung in tools/convert_hf_to_torch_dist.py, not in training. Four ranks reached the barrier in Megatron's dist_checkpointing execute_sync and rank 0's NCCL watchdog killed the job after the 600s default: WorkNCCL(SeqNum=9, OpType=ALLREDUCE, NumelIn=1) timed out #0 barrier torch/distributed/distributed_c10d.py:5030 #2 execute_sync Megatron-LM/.../strategies/async_utils.py:99 #7 main tools/convert_hf_to_torch_dist.py:131 Not a Blackwell problem: test_glm5_2_744b_a40b_5layer_nvfp4 passed in the same job, and nvfp4 kernels ran fine. It converts with --nproc-per-node 1, so it has no cross-rank barrier to time out on; this one uses 4. Not a slow disk either. The save wrote 22G of a ~50B-parameter MoE in 13 minutes, about 28 MB/s, while the host's NVMe measures 2.8 GB/s write and 3.2 GB/s read. Whatever stalls that path, raising the NCCL timeout only makes it hang longer. scripts/run_joy_ai_llm_flash.py stays -- it is a documented launch script and this only removes the CI wrapper that never worked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.