Move FT modules into topic folders (mechanical)

[fzyzcjy]

Group the fault-tolerance work into folders instead of scattering it
across miles/utils and megatron_utils:

• miles/utils/ft_utils/: control_server, health_checker, heartbeat_utils,
  indep_dp, mini_ft_controller, process_group_utils
• miles/utils/audit_utils/: event_analyzer, event_logger, witness,
  checksum_utils, process_identity
• miles/utils/tracking_utils/: structured_log
• miles/utils/test_utils/: clock, det_process_group
• miles/backends/megatron_utils/ft/: checkpoint_transfer,
  in_memory_checkpoint, indep_dp, types

The tracking facade moves from tracking_utils/init.py to
tracking_utils/tracking.py (callers import the submodule) so that
importing tracking_utils.structured_log does not run a package
init that circularly imports the event logger.

Pure relocation: git-mv moves, empty package init files, dotted
import-path rewrites, and dot-depth fixes for the relative imports
that cross a moved path (plus isort reordering). No code changes.

Script: ft_restructure_transform.py (attached to the PR); reproduce via
verify_mechanical_refactor(base, target, transform) — byte-identical.

ci-sglang-pr: #28524

[gemini-code-assist]

Code Review

This pull request reorganizes the codebase structure by refactoring and moving various utility modules into more structured subdirectories. Specifically, fault tolerance (FT) utilities have been relocated to miles/utils/ft_utils/, auditing and witness utilities to miles/utils/audit_utils/, tracking and logging utilities to miles/utils/tracking_utils/, and certain test utilities (like clock and det_process_group) to miles/utils/test_utils/. Megatron-specific fault tolerance modules have also been moved under miles/backends/megatron_utils/ft/. All corresponding imports and test files have been updated to reflect these new paths. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[fzyzcjy]

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the stage-c-8-gpu-h100 failures by comparing against main's last green nightly and PR #1559's own CI history. Please push back if any conclusion is off.

CI status note (2026-07-08):

stage-c-8-gpu-h100 failures here are pre-existing on main, not introduced by this chain:

• Failing tests: tests/e2e/megatron/test_glm5_2_744b_a40b_5layer_lora_ci.py (partition 1) and tests/e2e/megatron/test_glm5_1_744b_a40b_6layer_lora_ci.py (partition 0), both with ValueError: Invalid experimental attention variant: dsa raised from Megatron-LM experimental_attention_variant_module_specs.py.
• The exact same failures occurred on PR Add GLM-5/5.1/5.2 (744B MoE) LoRA RL support  #1559's own final CI run (https://github.com/radixark/miles/actions/runs/28896051695) with the default miles-main Megatron, and Add GLM-5/5.1/5.2 (744B MoE) LoRA RL support  #1559 was merged with that red. These GLM5 LoRA tests have never passed in a full run since merge; the dsa spec dispatch is missing from Megatron miles-main itself.
• Not caused by the ci-megatron-pr: #21 override: PR Refactor and extract RolloutHealthMonitor #21's head is byte-identical to miles-main in every dsa-related file, and the container image digest matches the last green nightly.

Fix: radixark/Megatron-LM#64 wires the existing (previously dead) get_dsa_module_spec_for_backend into the experimental-attention-variant dispatch. The fix is also merged into the ci-megatron-pr: #21 branch this PR pins, so rerunning the failed h100 jobs picks it up.

[fzyzcjy]

🤖 Posted autonomously by Claude Code acting on the user's behalf. Triaged the stage-c-4-gpu-h200 (0) failure from the job log and the last green nightly. Please push back if any conclusion is off.

stage-c-4-gpu-h200 (0) failure (https://github.com/radixark/miles/actions/runs/28921429924/job/85803894664):

• Failing test: tests/e2e/megatron/test_qwen3_30B_A3B/test_int4_rollout.py — not an FT test; it runs here because run-ci-image forces --match-all-labels.
• Fingerprint: train step 0 completed normally (ft op=train_step ... outcome=NORMAL), then one MegatronTrainRayActor process died hard: ray.exceptions.ActorUnavailableError: ... RpcError: RPC error: Socket closed rpc_code: 14 — an abrupt worker-process death, no Python traceback on the actor side.
• The same suite (including this test) passed on main's last nightly (run 28879970610, 2026-07-08) with the same container image digest, and partitions (1)/(2) of this stage passed here.
• Classification: suspected flake/infra (abrupt actor death after a healthy training step). Next step: rerun failed jobs once the run completes and compare.

[fzyzcjy]

🤖 Posted autonomously by Claude Code acting on the user's behalf. Follow-up on the stage-c-4-gpu-h200 (0) triage above after the rerun completed.

Rerun result: stage-c-4-gpu-h200 (0) passed on attempt 2 (same code, same image) — the test_int4_rollout.py abrupt actor death is confirmed a flake, as suspected.

[fzyzcjy]

🤖 Posted autonomously by Claude Code acting on the user's behalf. After moving the FT e2e suite to stage-c-8-gpu-h100, the FT tests ran in CI for the first time and surfaced a dependency gap. Reporting the root cause; how to resolve is a maintainer call.

Two independent failures on stage-c-8-gpu-h100, different root causes:

1. GLM5 LoRA tests (test_glm5_1/2_*_lora_ci.py) — pre-existing, now fixed.
dsa experimental-attention spec gaps in Megatron + a stale Megatron-Bridge in the radixark/miles:dev image. Fixed by radixark/Megatron-LM#64 (merged into the pinned ci-megatron-pr: #21) plus a fresh dev image rebuild (the scheduled rebuild had been broken by a transient 429). These should pass on the next run against the fresh image.

2. FT e2e tests (test_trainer_ft_*) — chicken-and-egg dependency, needs a decision.
test_trainer_ft_realistic_gsm8k.py fails at init:

ModuleNotFoundError: No module named 'nvidia_resiliency_ext'
RuntimeError: nvidia_resiliency_ext is required for local checkpointing

Root cause: miles auto-sets non_persistent_ckpt_type="local" whenever train is in --ft-components (miles/utils/arguments.py), and Megatron requires nvidia_resiliency_ext for local checkpointing. So every FT test needs it. nvidia-resiliency-ext is declared in requirements.txt — but only on this chain (added by "Add the fault-tolerance dependency, CI label, and logger-config setup"); it is not on main. The dev image is built from main, so it lacks the module. These tests never ran in CI before (they were registered on the non-existent stage-c-8-gpu-h200 suite), so the gap was invisible until this suite move.

Consequence: the FT tests can only go green in CI against an image built from a requirements.txt that includes nvidia-resiliency-ext — i.e. after this chain lands on main (nightly rebuild), or via a custom image built from the chain and pinned with ci-image-tag: for verification.

[fzyzcjy]

🤖 Posted autonomously by Claude Code acting on the user's behalf. Explaining why the run-ci-ft-short / run-ci-ft-long labels were dropped from this PR.

Why the FT tests can't run in CI yet (labels removed).

The FT e2e tests were moved to the stage-c-8-gpu-h100 suite (they were previously registered on a non-existent stage-c-8-gpu-h200 suite, so they never actually ran in CI). Running them for the first time revealed a hard dependency gap:

• Whenever train is in --ft-components, miles auto-sets non_persistent_ckpt_type="local" (miles/utils/arguments.py), and Megatron requires the nvidia_resiliency_ext module for local checkpointing. So every FT test needs it — test_trainer_ft_realistic_gsm8k.py crashes at init with RuntimeError: nvidia_resiliency_ext is required for local checkpointing.
• nvidia-resiliency-ext is declared in requirements.txt, but only on this chain (added by the "Add the fault-tolerance dependency…" commit). It is not on main.
• The radixark/miles:dev CI image bakes its Python deps at build time (COPY requirements.txt + pip install -r), built from main. Since main's requirements.txt has no nvidia-resiliency-ext, the image doesn't have the module — and the CI install step uses pip install -e . --no-deps, so it isn't pulled in at test time either.

Net: this is a build-order (chicken-and-egg) situation — the FT tests can only go green against an image built from a requirements.txt that includes nvidia-resiliency-ext, which happens once this chain lands on main and the nightly rebuilds dev. Forcing them to run now would only fail on the missing module, not on anything about the restructure.

So the run-ci-ft-* labels were removed from this PR (they were also being force-run by run-ci-image's --match-all-labels, which is why that label was dropped too). The restructure itself is verified by the mechanical byte-identical proof; run-ci-model-scripts is kept to confirm the GLM5 path (unrelated pre-existing dsa fix) is green on the freshly rebuilt image.

[fzyzcjy]

all green