initial - #505
Merged
Merged
Conversation
Oseltamivir
added a commit
that referenced
this pull request
Aug 31, 2026
…(mori >= #505) AsyncLL under the DSv4-Pro workload (topk 6) dies on every shipped mi35x-mori image: ROCm/mori#505 (AsyncLL slot assignment double-allocates when top-k does not divide warpSize, fixed upstream 2026-07-31) postdates them all. On-metal probe of this date-stamped nightly (job 41563, pure mori.ops): AsyncLL topk6@384 and topk8@256 both pass, split-phase recv API present. mi355x only — the -tw SKUs keep their image until separately validated on their docker path.
Oseltamivir
added a commit
that referenced
this pull request
Sep 9, 2026
… AsyncLL, DSv4-Pro workload (#2786) * CollectiveX: publish wire-basis bytes so LL bandwidth stops mixing byte definitions DeepEP/UCCL/NCCL low-latency kernels move one copy per (token, expert) assignment; normal mode and MoRI LL move rank-deduplicated (token, dest-rank) copies. byte_provenance always carried the smaller deduplicated numerator, so any bandwidth divided from it on a token-expert backend was a lower bound published as the wire rate (34.1% low on nccl-ep LL EP8 at T=128: 77.4 MB vs the 117.4 MB the kernels move) and not comparable across backends. Rows now carry wire_byte_provenance beside byte_provenance: identical for token-rank receives, per-assignment for token-expert ones, basis declared by the existing logical_copies.wire discriminator. bandwidth.py divides from the wire basis (falling back to the deduplicated one for pre-wire artifacts, which only ever understates). * CollectiveX: hold nccl-ep low-latency fleet-wide until the combine fence ships The T<=128 ladder clamp reduces exposure to the un-fenced shared-memory combine race (DeepEP #642's fence, absent at our pin and at NVIDIA master); the source has said all along it is not a safety boundary — lower rungs are less likely to hit the race, not immune. Green clamped rows are therefore not publication-valid, and describing T=256 as a capacity limit was wrong. Rows held in the registry, normal mode unaffected. * CollectiveX: charge nccl-ep's per-step routing collective to the timed HT dispatch ncclEpUpdateHandle is documented as a per-step collective and production routing changes every MoE layer, so a serving step pays it before every HT dispatch at the handle's full token capacity. Excluding it (as NVIDIA's ep_bench does) made HT dispatch the one window that omitted its routing work while deepep-v2, uccl-ep, MoRI and FlashInfer all carry theirs per call. kernel_generation bumps to nccl-ep-ht-routed — the per-row discriminator the earlier NCCL measurement changes lacked. * CollectiveX: measure MoRI low-latency on AsyncLL, the kernel production deploys SGLang's moriep dispatcher maps low-latency to AsyncLL split-phase (dispatch_send/recv + combine_send/recv, block_num 64 rdma 32 warps 8) and asserts that kernel type; IntraNodeLL is not selected by any engine, so the previous rows measured an off-production path (kernel_generation intranode-ll discriminates them). The timed windows now run AsyncLL send+recv back-to-back and the adapter fails closed on wheels whose AsyncLL lacks the recv API — single-call AsyncLL returns before any payload lands. Pending on-metal validation on mi355x. * CollectiveX: sync methodology with the wire basis, the nccl-ep LL hold and the routed HT window * CollectiveX: switch the EP workload to DeepSeek-V4-Pro routing deepseek-v4-pro: hidden 7168, top-k 6, 384 routed experts — the dsv4 shape the rest of InferenceX benchmarks (operatorx testlists, HF-config-derived). The workload name is part of every case_id, so this is a clean identity break in the durable store: no v3 row can collide with a v4-pro row. Ladders and seed unchanged. * CollectiveX: sync the README's MoRI low-latency kernel description with AsyncLL (review) * CollectiveX: rewrite the payload-bandwidth section around the wire basis (review) The Measurement section still described the rank-deduplicated count as the sole GB/s numerator and cited the retired IntraNodeLL as the dedup exemplar; bandwidth.py and the app now divide from wire_byte_provenance. The section now names both accountings, which rows differ, the 34% understatement that motivated the change, and the fallback behavior for pre-wire artifacts. * CollectiveX: exclude b300-001 and b300-016 (mlx5_4 DOWN) The full v4-pro sweep's two b300 EP16 legs each drew one of these nodes and failed closed at the network-profile gate (rdma-port-5=inactive = the 5th selector, mlx5_4). Idle-pool survey: exactly these two nodes have mlx5_4 port state 1: DOWN; the other 12 idle nodes are 4: ACTIVE — a sick node pair (likely one leaf/cable pair), not a rail-wide outage. SRE handback owed. * CollectiveX: hold gb300 low-latency until the nvshmem-over-MNNVL init regression is resolved All gb300 deepep-v2 LL shards die at the first CUDA op after Buffer init with cudaErrorUnknown (context poisoned during nvshmem symmetric-heap init over MNNVL), on every rank, both degrees and precisions. Discrimination: gb300 normal mode green at n2/n4; gb200 LL green with the same image/pin/workload; a main-branch control with the old workload fails identically — so neither this PR nor the v4-pro shape. CI-shaped 2-node hand probe (job 27430 on the pool) reproduces: bootstrap and buffer creation complete, then every rank's next CUDA call returns unknown error. Driver is 580.159.03 (installed in the KV-fix window ~2026-08-26..28, after the last green gb300 LL run). Platform escalation owed; rows held so sweeps don't burn four known-red shards each. * CollectiveX: bump the mi355x image to v0.5.18-rocm720-mi35x-20260830 (mori >= #505) AsyncLL under the DSv4-Pro workload (topk 6) dies on every shipped mi35x-mori image: ROCm/mori#505 (AsyncLL slot assignment double-allocates when top-k does not divide warpSize, fixed upstream 2026-07-31) postdates them all. On-metal probe of this date-stamped nightly (job 41563, pure mori.ops): AsyncLL topk6@384 and topk8@256 both pass, split-phase recv API present. mi355x only — the -tw SKUs keep their image until separately validated on their docker path. * CollectiveX: build DeepEP against the image-matched nvshmem (cu13) — un-breaks gb300 LL The gb300 LL cudaErrorUnknown was ours, not the platform's. 2x2 on the pool: stock in-image deep_ep (built on nvidia-nvshmem-cu13==3.4.5) runs LL clean on the exact r01 nodes and driver (580.159.03) where the CI stack dies; the CI venv's nvidia-nvshmem-cu12==3.3.9 r12 host library survives sm90/sm100 but poisons the CUDA context during symmetric-heap init over MNNVL on sm103 — buffer creation returns, the next CUDA call on every rank fails. The earlier driver-window attribution was a coincidence of timing (pin bump and driver install overlapped; hold commit e9ef099 superseded). The wheel spec now lives in common.sh (COLLX_DEEPEP_V2_NVSHMEM_SPEC), is installed by prepare_backend, and keys the venv cache so every NVIDIA deepep venv rebuilds once against cu13. gb300 ll_backends restored. * CollectiveX: torch 2.11.0 for the DeepEP venv — the actual gb300 LL fix Root cause, finally isolated by holding everything else constant: a venv identical to prepare_backend's recipe (same DeepEP pin, same nccl-cu13 2.30.4, same nvshmem-cu13 3.4.5) is green with torch==2.11.0+cu130 and red with torch==2.10.0+cu130 on the same gb300 r01 nodes (jobs 27760 vs 27430/27757). torch 2.10.0's bundled CUDA-13 userland poisons the CUDA context during nvshmem symmetric-heap init over MNNVL on sm103 + driver 580.159.03; sm90 and sm100 are unaffected. Falsified along the way, in order: the v4-pro workload (main control), the driver alone (stock image green on the failing nodes — the user's r03 canary forced that 2x2), the nvshmem cu12 wheel (cu13 rebuild still red), the DeepEP pin (old-pin venv red). 2.11.0 is what the cu130 image ships. The torch spec joins the nvshmem spec in the venv cache key, so every deepep venv rebuilds once. * CollectiveX: pin the RDC device-link arch — the real gb300 LL root cause The DeepEP build compiles every object with the right -gencode (setup.py 'Arch list: 10.3', all compile lines sm_103), but the final nvcc -dlink step receives NO gencode and falls back to nvcc's CUDA-13 default, sm_75 — the .so's single linked device image is sm_75 and its kernels can never load on the target GPU. On gb300/sm103 that surfaced as cudaErrorUnknown at the first legacy-kernel launch (LL and layout paths; ElasticBuffer JIT paths were unaffected, which is why normal mode stayed green). Proven by instrumented runs of prepare_backend's own build (job 27792: compile lines sm_103, step 9/9 -dlink bare, product sm_75) after hand builds of the identical recipe produced sm_103 and ran LL clean on the failing nodes. Fix: NVCC_PREPEND_FLAGS carries -gencode=arch=compute_<arch>,code=sm_<arch> into every nvcc invocation including the dlink. COLLX_DEEPEP_V2_BUILD_GEN (dlarch1) joins the cache key so the sm_75 venvs with valid .ready markers rebuild instead of being reused. The torch 2.10->2.11 bump (previous commit) stays: 2.11 matches the image and the 2.10-vs-2.11 A/B was run against differently-linked binaries, so its verdict is voided rather than reversed; the pin question can be revisited upstream of this fix if anyone cares to re-test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.