[DO NOT MERGE] CI 1 - #2407
Conversation
fzyzcjy
added
run-ci-image
bypass-fastfail
The This is an external download-rate-limit failure rather than a Miles assertion failure, as shown by the same job log. GPU siblings are already active in the PR checks, so I will leave them undisturbed and rerun the failed job after the workflow settles. |
The completed GPU failures on head All four logs have the same primary fingerprint: Classification: our code bug, fixed on the new head. Root cause: Fix: commits Pre-push validation on a 1x H200 devbox:
The new final-head CI is now running on |
|
CI audit update (AI-assisted by Codex)
|
|
Final CI audit (AI-assisted by Codex)
|
a58b3f6 to
0963e0e
Compare
|
Final CI audit for head 0963e0e:
Final rollup: 40 successful checks, 2 skipped approval-holder checks, no pending or failing checks. PR is MERGEABLE/CLEAN. |
53e2412 to
b27e9a0
Compare
Green on
|
| Lane | Test | Attempt-1 fingerprint | Attempt-2 |
|---|---|---|---|
stage-c-4-gpu-h200 (1) |
test_dp_attention.py |
gloo Connection closed by peer in _send_to_colocated_engine |
PASS elapsed=763s, 9/9 tests |
stage-c-4-gpu-h200 (2) |
test_r3_baseline.py |
trainer rank0 vanished with no output; TCPStore EOF + raylet SYSTEM_ERROR |
PASS elapsed=941s, 9/9 tests |
stage-c-8-gpu-h100 (0) |
test_r3_mtp.py |
Cuda failure 999 'unknown error' via NCCL inside TransformerEngine CP attention |
PASS elapsed=802s, 10/10 tests, CI Gate TRUSTED (18 metric(s)) |
Attempt-1 jobs: 4-gpu (1), 4-gpu (2), 8-gpu-h100 (0).
Why these read as infra rather than regressions:
- None of the three raised from miles Python logic. Two are a peer training process disappearing with no self-reported error (the gloo FIN and the TCPStore EOF are two transports noticing the same class of event; raylet's own message lists OOM-killer / SIGSEGV as the usual causes). The third is a CUDA driver error surfacing inside third-party TransformerEngine/NCCL code right after a sleep/wake GPU memory reload.
test_dp_attentionfailed in code this branch did not change since a green run. The same lane passed at0963e0efc, andgit diff 0963e0efc..4802a93e9a -- miles/backends/megatron_utils/update_weight/is only the two-line mooncake change above.reloadable_process_group.py, on the causal path of the CUDA-999 failure, has zero commits indeliver-1-base..HEAD.- Sibling tests in the same jobs passed, including other tests exercising the same wake_up / colocate pattern.
- Where this branch is in the blast radius — the new
RayWorkerHandledeath-confirmation path andTrainerCellerror handling — it behaved correctly: it detected the dead worker, marked the cell errored, and failed fast with a clearWorkerUnreachableErrorrather than hanging or silently corrupting state.
Residual uncertainty worth stating: GitHub Actions logs carry no host dmesg, so a kernel-level OOM-kill of the vanished processes can be neither confirmed nor excluded. If the "silent trainer process death" signature recurs on this branch, host memory pressure is the first thing to check.
2f5a8ba to
1530b0b
Compare
Head under test
Two duplicate runs on the same SHA; the no-coverage one is cancelledTwo
The CPU gate is green, and it really executed tests
A green GPU/CPU job can still be a no-op, so each was checked for a non-zero
The four The nine Dependency pin is currentThe PR body pins
Devbox pre-verification of the same head: zero real failuresRun on an H200 devbox with
All 280 errors in pass 1 collapse to one cause — Caveat: a green devbox is not evidence for a green CPU runner. The hosted CPU runner has no accelerator and roughly 16GB of RAM, so accelerator-probing and resource-ceiling failures cannot appear on a devbox. The Note on the earlier queueingThese CPU stages sat |
0dce880 to
e6999a7
Compare
Head under test
The chain now sits on upstream main CPU gate is green
The previous run on this chain had six CPU failures. All six came from three integration gaps left by the rebase onto new main, each fixed in the commit that owns it (no new commits, chain invariant preserved):
Each fix landed in Two GPU failures remain, and both are upstream defects this rebase inherited1.
|
The rebase onto upstream main introduces zero test regressionsSame command, same devbox, same pinned sglang/Megatron, same
Counts alone prove nothing, so the sets were diffed by test id:
The one "introduced" failure is a test-environment artifact, not a regression
So the attribution is introduced = 0, which matches what the rebase report claimed on static grounds. The 320 errors on both sides are the known consequence of blanking CI status on
|
Per the review decision, FT test actions now name their target cell with a full cell id instead of a positional cell_index, superseding the index-based addressing that had to be resolved at runtime. - FTTestAction drops cell_index (and its -1 "last cell" convention) in favour of a required cell_id; resolve_cell_id is gone. - Loading validates every cell_id parses, so a typo fails at startup instead of silently never matching. - The group executor asserts the target's spec name and cell index fit the group before dispatching, then forwards the cell id verbatim to stop_cells/start_cells; no runtime num_cells denominator is involved. - The actor executor no longer takes a cell_ids roster and matches by plain string equality on cell_id and rank. - The two e2e FT scenarios build the cell id from their statically known num_cells, and the --ci-ft-test-actions help text documents the new shape. Tests not executed (test run ban); statically checked only.
RayWorkerHandle mirrors RpcWorkerHandle: keyword-only magic dispatch that translates actor death into WorkerUnreachableError. wait_dead joins the handle contract so callers that kill a worker can confirm it is gone without knowing the transport; the rpc side has no platform death signal yet and says so.
The checkpoint conversion step in all three GLM guides still calls scripts/model_args.py, which no longer exists, so anyone following the recipe gets a file-not-found before the conversion starts.
…celerators A CPU suite runs many files in one pytest process, so whichever file calls ray.init first fixes the cluster's resources for every later file: a second initializer produces a cluster without the logical GPUs the placement-group tests need. Route the remaining callers through the session-scoped ray_local_mode fixture and add a test that keeps tests/conftest.py the only caller. Also reap the accelerator processes a suite leaves behind, give the mock engine the inject_fault the real CommandActor has, and fix the tests that were asserting shapes main had already moved past.
The train actor used to tell the rollout side which weights the engines now serve via `self.rollout_manager`, but that attribute stopped being assigned once the actor was decoupled from the rollout side, leaving two dead call sites: the FSDP one also referenced an unimported `ray`. The executor's `weight_version` therefore stayed None forever and every sample it collected went to training unstamped. The actor's `update_weights` now answers the version it broadcast, the trainer controller passes it up, and the driver writes it into the executor through one `update_weights` helper the three training loops share. An actor that skipped the broadcast answers None and nothing is published. The driver owns the write because it is the only place that holds both handles for the whole chain: the controller's own executor handle goes away when the trainer controller becomes an independent worker.
The executor only learns the served weight version when the driver forwards what TrainerController.update_weights returns. A training script that calls update_weights directly keeps training fine while the executor's version stays frozen, which silently disables --max-weight-staleness accounting. Count rollouts served since the last publish and assert the count stays small, so the broken wiring fails loudly within a few steps instead of never.
The api server ran on a daemon thread through uvicorn.run, so a startup failure only produced a traceback nobody watched. A second concurrent job on the same host then polled 127.0.0.1:18080 and read the first job's cell registry, and the two runs silently drove each other's cells. Wait for uvicorn to report started and re-raise the failure on the caller thread. The wait itself is a small private helper, since a daemon thread that dies alone is invisible to whoever started it.
The reset_tensors check ran after the cell had already moved to PendingWeights. A failing check raised into the tick loop, which only logs, and no later tick retried it because the state had moved on. Readiness accepts PendingWeights, so a check the user explicitly asked for was silently skipped, and the run could even pass on tensors that happened to match because they were never poisoned. The check hung off an elif of the serve-without-weight-update branch, which tied it to a condition it does not depend on. Spell that condition out as its own if and move the call above the transition, so a failure leaves the cell Initializing and the next tick retries it.
StateInitializing had no failure exit: _tick_when_initializing returned on a failed health probe, the health checker only runs in PendingWeights and Serving, and an initializing cell reports Pending with no Healthy condition, so the mini ft controller read NotApplicable and skipped it. WorkerManager.alive reads a local actor list that a dead subprocess never updates, so nothing healed the cell either. A replacement whose engine died during startup stayed stranded until the next hour-long readiness timeout, long after the evidence was gone. Stamp the startup time on StateInitializing, and report Healthy=False with reason StartupDeadlineExceeded once a cell sits there past the deadline. A false Healthy verdict already outranks phase in the mini ft controller, so it heals the cell through the normal channel, as does any external controller reading the same status.
MilesRouter only served /add_worker and /list_workers, so ServerCell.dispose hit the catch-all proxy and its error was swallowed. Every cell generation appended a fresh URL to worker_request_counts, worker_failure_counts and dead_workers and nothing ever removed one. list_workers kept returning dead workers, and the synchronous class-based rollout aborts against the full list every step, paying 60 retries per dead address before the gather finally raises. Add /remove_worker with the same URL conventions as /add_worker, and share the URL parsing between them.
All three calls at the top of EvalFleet.pin were broken: RolloutServer no
longer has recover() or wait_all_engines_alive(), and
ServerCell.probe_and_mark_dead() referenced is_allocated/_mark_stopped,
which no longer exist, so it raised AttributeError on every call. The
surrounding try/except swallowed the failure into EvalSkip("unhealthy"),
so every eval-fleet pin silently degraded to a skipped eval.
The eval fleet does not support fault tolerance, so the path is removed
rather than reimplemented: pin now goes straight to the weight load, and
the now-unreachable RolloutServer.probe_and_mark_dead and
ServerCell.probe_and_mark_dead are deleted along with their tests.
66c26ce to
97bba45
Compare
CI status after rebasing the chain onto latest mainHead is now Result: 24 success / 2 failure / 9 skipped. Every CPU lane, Runs: PR Test #31774484373 · PR Test (ROCm) #31774484412 Green lanes, with durationsDurations are included because a fast "success" can mean the lane never actually ran. These are real runs.
The two red ROCm shards
Every failing test in both shards fails the same single way, with no second failure mode: the sglang engine starts and serves normally ( Root cause: the ROCm image's sglang has no gated-launch supportThe decisive evidence is sglang's own
The sibling port arrives while the gate port does not, so the argument plumbing on the miles side is fine — that sglang build simply does not have the field. Both Why this is not introduced by this branch
Three possible ways forward, none of them applied here since the ownership is not obvious: rebuild the ROCm CI image on an sglang that has gated launch; teach miles to probe backend capability and take the existing no-gate path ( |
1 participant
ci-sglang-pr: tom/refactor-miles-repo-sglang/deliver-1
Replacement for #2196, which was accidentally merged as a zero-commit PR while the base ref briefly pointed at the head tip during chain absorption.