[ZhenlongMa]

What?

Add a stats-enabled unit test for multi-rail load-balance after a device speed-change event. The test verifies that PUT and GET operations can distribute and reallocate transmission data across two rails based on their actual rates.

Why?

The existing unit test for speed-change only verify the configuration without checking the actual data on each rail. As a result, we need additional tests to guarantee that UCX adopt the latest and correct multi-rail speed distribution.

How?

The test records the byte counters for every lane before and after the data transmission and compares them with the expected distribution.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

the per-lane split is computed with round-up fixed-point weights (ucp_proto_multi_scaled_length() uses div_round_up), so the actual bytes on each device are not the exact expected_bytes * rate / total_rate value used here. is a 1-byte tolerance robust across the possible weight rounding, or could it be flaky? if it depends on the specific 52K/38K message sizes, maybe add a short comment noting why these sizes were chosen.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (WireCompat Test wire compatibility althca ucx_1_17) · commit fbfab736

TL;DR: The wire-compat job failed in the compare tls performance characteristics step because the PR-built (v1.24) UCX artifact has no cma and knem transports at all, while the locally-built v1.17 reference does — an artifact/build-environment mismatch (the cma/knem UCT modules were not built into the published artifact), not anything from PR #11842.

Full analysis

Summary: buildlib/tools/check_tls_perf_caps.sh exited 1 with Fail: (memory/cma/bandwidth) 0.00/ppn + 11145.00 MB/sec != and the same for memory/cma/overhead, memory/knem/bandwidth, memory/knem/overhead (right-hand side empty).

Root cause: The script compares ucx_info -d of the legacy build (old value, left) with the PR build (new value, right). In the log, the PR-side ucx_info -d dump ends after Memory domain: posix — it contains only self, tcp, sysv, posix; there is no cma or knem memory domain/transport, so new_cap is an empty string for those and the comparison fails (check_tls_perf_caps.sh:47-50). All transports that exist on both sides match exactly (self 19360, tcp 113.16/11.91, sysv/posix 15360).
Why they're missing: cma and knem are dlopen'd UCT modules (uct_modules="${uct_modules}:cma" in src/uct/sm/scopy/cma/configure.m4:19, :knem in src/uct/sm/scopy/knem/configure.m4:23), enabled only if configure detects sys/uio.h/process_vm_readv and knem_io.h. The legacy library is configured/built in the test job on the test host (buildlib/pr/wire_compat.yml:75-90), while the "current" library comes from a separate build_pr_althca job on a different agent and is consumed as a published artifact (lines 34-46, 69-74). That artifact evidently was configured without CMA/KNEM, so lib/ucx/libuct_cma.so / libuct_knem.so are absent. Module loading from the artifact itself is fine — the same 1.24 library successfully loads the rdmacm module later in the log (rdmacm_cm.c:953 messages), so this is a build-configuration/agent-capability gap, not a permissions or dlopen problem. PR #11842 only adds a gtest for multi-rail load balance and cannot change ucx_info -d output, so this is a CI-infrastructure failure, not a code regression.

Implicated commit: unknown (no code commit implicated; environment/build-config difference between the build_pr_althca agent and the swx-rdmz-ucx-althca test host)

File: buildlib/tools/check_tls_perf_caps.sh:29-56 (comparison loop) with buildlib/pr/wire_compat.yml:34-46 (artifact build job) as the source of the mismatch

Suggested fix:

1. Verify first: in the test job add ls -l $(System.DefaultWorkingDirectory)/$(ucx_current)/lib/ucx/ and dump ucx_info -v (configure line) for both builds — expect libuct_cma.so/libuct_knem.so to be missing from the current artifact.
2. Make the mismatch impossible: build the PR library with explicit requirements in build_pr_<name> (./contrib/configure-release --enable-cma --with-knem ...) so configure fails loudly on an agent lacking knem headers, and/or install knem-devel/knem pkg-config on the althca build agents. Alternatively give the build_pr_${{ parameters.name }} job the same demands / host class as the test job (or build the current library in the test job, like the legacy one) so both are configured against identical system capabilities.
3. Harden the checker so this reports a useful message instead of X != : if a legacy transport/device block is absent from the PR output, emit e.g. transport cma not present in PR build (module not built?). Also consider replacing the fragile fixed grep -A 8 / -A 7 windows with block-aware parsing.

Related: PR #11842 (the innocent PR under test); #11720 / #11654 (recent CMA probe changes — worth ruling out with UCX_LOG_LEVEL=debug ucx_info -d on the test host if the modules turn out to be present in the artifact after step 1)

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

After a speed change the EP is reassigned to a freshly created ep config, and expect_byte_distribution reads ucp_ep_num_lanes(ep) from that new config while bytes_before was sized from the config in effect at capture time. If a future variant of this test (or a config where a rail is dropped/added on speed change) makes the new config have more lanes than bytes_before.size(), bytes_before[lane] reads past the end of the vector. In this specific mock the rail count stays fixed at 2, so it's safe today — worth a short comment noting the assumption, or sizing/reading bytes_before defensively.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Inconsistent null-lane handling / potential crash: expect_byte_distribution() skips lanes whose rsc_index == UCP_NULL_RESOURCE (L2491), but get_zcopy_bytes_each_lane() (and get_zcopy_bytes_sent()) iterate over every lane 0..num_lanes and unconditionally dereference ucp_ep_get_lane(ep, lane)->stats. If the ep config contains a lane with UCP_NULL_RESOURCE (e.g. a wireup/CM lane — existing code such as test_ucp_rma.cc:408 guards for exactly this case), the collection helper reads a null/invalid uct_ep and can crash before the guard in expect_byte_distribution is ever reached. Suggest applying the same null-resource guard here so a wireup/CM lane doesn't crash the collection. Non-blocking: in this specific rc_x-only mock config (no CM lane, IB_NUM_PATHS=1, MAX_RMA_RAILS=2) the RMA lanes likely all have valid rsc_index, so the crash may not trigger in practice, but the inconsistency with the guard the author already added warrants alignment.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests roce on worker 2) · commit 62ac4d76

TL;DR: The roce on worker 2 gtest run failed on a single unrelated test, ib/test_md.alloc/1 (mlx5_1), which is the memory-pressure-sensitive UCT IB MD allocation test — nothing in PR #11842 (a new UCP multi-rail load-balance gtest) touches UCT/IB MD code, so this is a flaky/environmental failure that should be retried.

Full analysis

Summary: make test in build-test/test/gtest exited with error 1 because 1 of 8721 tests failed: [ FAILED ] ib/test_md.alloc/1, where GetParam() = mlx5_1; all other 8720 tests passed.

Root cause: From the log evidence: the whole 60-minute gtest run (3633567 ms total) is clean except this one UCT MD allocation test on the IB (mlx5_1) memory domain. test_md.alloc hammers uct_mem_alloc() with UCT_ALLOC_METHOD_MD 300 times per memory type plus indirect/atomic mkey packing (UCT_MD_MKEY_PACK_FLAG_INVALIDATE_RMA|AMO), and it only tolerates UCS_ERR_NO_MEMORY up to a 50 % failure rate (EXPECT_LT((double)num_alloc_failures / iterations, 0.5) << "Too many OUT_OF_RESOURCE failures"); any other status trips ASSERT_UCS_OK(status). The same worker/host is concurrently running tests that consume multi-gigabyte buffers (visible in the same log: exp_huge_mix ... 1073741856 bytes, self/uct_p2p_rma_test ... 4402341478 bytes) across parallel gtest workers, i.e. the machine is under heavy host-memory/device-resource pressure at that moment — exactly the condition this test is known to be fragile under (previously patched by PR #10150 "TEST/MD: Add short sleep before retrying memory allocation"). The PR under test (#11842, "UCP/GTEST: Add test for multi-rail load-balance") adds only a UCP-level gtest and cannot influence IB MD allocation, and no recent commit touches test/gtest/uct/test_md.cc (last: 272c989, Aug 2026, ZE-only) or the IB MD alloc path in a way tied to this branch.

Caveat: Azure only returns the tail of this job's log, so the exact gtest assertion text/line for ib/test_md.alloc/1 (which of ASSERT_UCS_OK vs. the "Too many OUT_OF_RESOURCE failures" expectation fired) is not visible in the retrievable portion.

Implicated commit: unknown — not attributable to [REDACTED:Hex High Entropy String] (PR #11842 adds a UCP multi-rail load-balance gtest only); most recent related test change is 272c989 (Yaser Afshar), unrelated to the IB path.

File: test/gtest/uct/test_md.cc:413-496 (failure point is either ASSERT_UCS_OK(status) at :475 / mkey pack at :487, or the EXPECT_LT(... < 0.5) << "Too many OUT_OF_RESOURCE failures" at :493)

Suggested fix:

1. Re-run the roce on worker 2 job on this PR — the failure is unrelated to the diff and expected to pass on retry.
2. If it recurs, harden the test rather than the PR: in test_md.cc, when uct_mem_alloc() returns UCS_ERR_NO_MEMORY for a high fraction of iterations, UCS_TEST_SKIP_R("host out of memory") instead of failing (extend the retry logic introduced in TEST/MD: Add short sleep before retrying memory allocation #10150), and/or lower iterations/max_size (currently 300 × up to 63356 B, line 415/447) for the IB MD variant.
3. Longer term, reduce co-scheduling of the multi-GB tests (uct_p2p_rma_test, test_ucp_tag_match_rndv.exp_huge_mix) with test_md.alloc on the same worker to cut memory-pressure flakiness.
4. To confirm the exact assertion, grab the full make test output for this job (the section around [ RUN ] ib/test_md.alloc/1) and attach it — the Azure API tail truncates it.

Related: PR #10150 (TEST/MD: Add short sleep before retrying memory allocation — prior fix for this same flaky test), PR #11842 (the PR under test)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id e8dd66ee-d90e-4623-9b79-5a07fff81596 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 3) · commit f0968f7d

TL;DR: The ASAN/roce shard-4 gtest run produced no output for 8m26s right after starting its first test (rc/test_ucp_rma_reg.put_blocking/1 <rc_v,cuda_copy,rocm_copy/nb>) and was then killed externally by an Azure agent shutdown — there is no test failure, assertion, or sanitizer report, and the PR's changes are gtest-only files unrelated to that fixture, so this is an environment/agent kill on top of a stalled test setup, not a code regression from f0968f7.

Full analysis

Summary: Job "roce on worker 3" (AddressSanitizer, gtest shard 4 of 4) was cancelled by ##[error]The agent has received a shutdown signal while the very first test in the shard was still running, with zero output for the preceding 8.5 minutes.

Root cause: Evidence from the task log:

• 18:07:07 — build is a no-op rebuild, 18:07:07.4 gtest launched with LD_PRELOAD=libasan.so.6, --gtest_filter=*, shard 4/4.
• 18:07:12.77 — [ RUN ] rc/test_ucp_rma_reg.put_blocking/1 <rc_v,cuda_copy,rocm_copy/nb> — last application output.
• 18:15:58.92 — The agent has received a shutdown signal…, then The operation was canceled.

The largest inter-line gap is 8m26s (≈95 % of the gtest runtime), so the process was not making progress: test_ucp_rma_reg::test_reg() → test_message_sizes() prints a <INFO host->host NxSIZE …> progress stream with an explicit fflush(stdout) (test/gtest/ucp/test_ucp_rma.cc:163-198), and stdout is unbuffered (stdbuf -o0). Since not a single size token was printed, the stall happened in fixture setup — test_ucp_memheap::init() → ucp_test::init() / sender().connect(&receiver()) (EP creation/wireup on rc_v with REG_NONBLOCK_MEM_TYPES=host) — before any transfer started.
The gtest watchdog would only have fired at 900 s (watchdog_timeout = 900. in test/gtest/common/test_helpers.cc:31), i.e. at ~18:22, so the agent shutdown at 18:15:58 pre-empted it and no backtrace/abort diagnostics were produced. The termination itself is agent/host-level (agent service stopped or container reclaimed), not a UCX exit code or timeout, and it is not attributable to the PR: the branch commits (3c5fc94, f0968f7 — Zhenlong Ma, "Add test for multi-rail load-balance" / "test code optimization") touch only files under test/gtest/ucp and do not touch ucp_test.{h,cc}, test/gtest/Makefile.am, or test_ucp_rma.cc.

Implicated commit: unknown — no product-code commit implicated; PR head is f0968f7 (Zhenlong Ma), whose changes are gtest-only and do not touch the stalled fixture.

File: test/gtest/ucp/test_ucp_rma.cc:748 (UCS_TEST_P(test_ucp_rma_reg, put_blocking), variant /1 = nb), stalled in setup via test/gtest/ucp/test_ucp_memheap.cc:16-20

Suggested fix:

1. Re-run "roce on worker 3" — the terminal event is an agent shutdown, an infra kill; if it passes, close as a CI flake (no code change needed for this PR).
2. If the silence reproduces, treat it as a real setup hang, not a slow test: capture gstack/gdb -p of the gtest process (or run just --gtest_filter='rc/test_ucp_rma_reg.put_blocking/1' under ASAN on swx-rain04) to see whether it is blocked in ucp_ep_create/wireup progress on rc_v.
3. Make future occurrences diagnosable instead of silent: lower the gtest watchdog for the ASAN job (e.g. ucs::watchdog_timeout / WATCHDOG_GTEST_TIMEOUT_ well under the agent's tolerance) so a hang aborts with a backtrace rather than dying with the agent, and consider having the test print the <INFO …> header before entering test_xfer. Do not raise the 200 m timeout — the job died at ~9 minutes, so the limit is irrelevant.

Related: PR #11842 (#11842) — the PR under test; no existing issue found for this hang/agent-shutdown signature.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.