[iyastreb]

What?

Raise UD_TIMEOUT from 1s to 3s in stale_private_ep_reuse and private_ep_keepalive.

Why?

CI hit ud_ep.c: Fatal: UD endpoint ... unhandled timeout error in stale_private_ep_reuse on the ASAN stage. The failing job stalled for ~2s (other UD tests in it ran 2s slower than in every other ASAN job). With a 1s UD timeout, a stall while a CREQ is in flight makes the UD timer declare a peer timeout on a user endpoint before the already arrived CREP is processed. The UD gtest fixture has no error handler, so that is fatal and kills the whole shard. 3s matches test_ud_peer_failure and survives such stalls; the keepalive idle window grows accordingly.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Review complete — no issues found.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 8c870e22

TL;DR: The ASAN roce job aborted in shm_gga/test_ucp_tag_nbx.basic/4 <shm,gga/iov> because a UMR mkey registration on mlx5_1 hung and hit the hard-coded 30 s ibv_poll_cq(UMR CQ) timeout; the resulting rkey-pack failure is not propagated, so an RTR was built with a non-zero address and zero-length rkey and tripped the fatal assertion in proto_rndv.c:883. This is unrelated to PR #11920 (UD test hardening) — retrigger, and fix the error path so pack failure aborts the request instead of asserting.

Full analysis

Summary: gtest crashed (core dumped, make: *** [Makefile:4713: test] Aborted) on shm_gga/test_ucp_tag_nbx.basic/4 <shm,gga/iov> with Assertion '(address != 0) == (rkey_length > 0)' failed: rtr rts->address=0x7f121b80f800 rkey_length=0.

Root cause: Log evidence, in order:

1. 14:40:01.98 last progress line, then a 30 s silence — exactly the hard-coded UMR poll timeout in uct_ib_mlx5_devx_umr_post_sync(): mlx5_1: ibv_poll_cq(UMR CQ, registration, UMR mkey 0x60400105c710 index 0x3025) timed out (ib_mlx5dv_md.c:1443). The UMR WQE never completed — the UMR QP was stuck (hang, not slowness; there was no other activity during that window).
2. The cleanup then posts an invalidation on the same (now broken/error-state) UMR QP: ibv_post_send(UMR QP, invalidation, ...) returned 12: Bad address (ib_mlx5dv_md.c:1434).
3. Registration failure surfaces as failed to pack remote key: Input/output error at proto_common.inl:446, which merely returns 0 (line 448) instead of failing the request.
4. ucp_proto_rndv_rtr_pack_with_rkey() has already written rtr->address = dt_iter->type.contig.buffer (rndv_rtr.c:149) before packing the rkey, so the RTR is sent with address != 0, rkey_length == 0. On the peer side ucp_proto_rndv_send_start() → ucp_proto_rndv_check_rkey_length() (proto_rndv.c:967/883) fatally asserts.

So: an environment/UMR-level hang on mlx5_1 (host swx-rain03) is the trigger, and a missing error path in the GGA/IOV rndv rkey packing turns it into a hard abort. Nothing in the failing path (gga_mlx5 UMR / rndv RTR) is touched by the branch iyastreb/ud-test-hardening.

Implicated commit: unknown — not commit [REDACTED:Hex High Entropy String] (PR #11920, UD tests only). Nearest owners of the implicated code: a9568621 (Roie Danino, UCT/IB relaxed-only memory keys) for the UMR md path, d65ffe7b (Michal Shalev, SGL/IOV) for the pack path.

File: src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:1441 (UMR poll timeout/hang), src/ucp/proto/proto_common.inl:445-449 (pack failure swallowed), src/ucp/rndv/rndv_rtr.c:149-161 (address set before rkey pack), assertion at src/ucp/rndv/proto_rndv.c:883

Suggested fix:

1. Retrigger the job — the UMR CQ hang is not caused by this PR; if it recurs on swx-rain03/mlx5_1, escalate the stuck UMR QP to the IB/GGA owners (a UMR WQE that never completes plus EFAULT on the follow-up post indicates the UMR QP went to error state).
2. Make the failure non-fatal instead of an assert: in ucp_proto_rndv_rtr_pack_with_rkey(), pack the rkey first and, if ucp_proto_request_pack_rkey() returns 0, set rtr->address = 0 (keeping the header self-consistent) or fail/complete the request with UCS_ERR_IO_ERROR rather than emitting an inconsistent RTR.
3. In uct_ib_mlx5_devx_umr_post_sync(), on UCS_ERR_TIMED_OUT mark the UMR QP as broken and recreate it rather than posting the invalidation WQE on it (avoids the secondary Bad address error), and consider making the 30 s timeout a config knob (it is much too long for a CI ASAN run, which masks the hang as a 30 s stall).

Related: PR #11920 (this build); no existing issue found for the UMR CQ timeout signature — worth opening one referencing Azure build 134960.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 08c4f528-39ec-44df-96a3-15c6bd8b2ce5 in the triage console for the audit trail.

[tvegas1]

maybe needs backport for stable tests in v1.23.x?

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 2) · commit 63317790

TL;DR: The ASAN "roce on worker 2" gtest run failed on a single unrelated test — dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1 (atomic fetch-add over device/MEMIC UCS_MEMORY_TYPE_RDMA memory on mlx5_1:1) — while all 8656 other tests passed; nothing in this UD-test-hardening PR touches DC/mlx5 atomics, so this is an infrastructure/flaky failure in the DC atomic-key-on-device-memory path, not a PR regression. Re-run the job and, if it reproduces, capture the full gtest output for that single fixture.

Full analysis

Summary: make test in build-test/test/gtest exited with error 1 because exactly one gtest case failed: dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64/1, where GetParam() = dc_mlx5/mlx5_1:1.

Root cause: From the log evidence: 8657 tests ran (2462952 ms ≈ 41 min of continuous activity — no stalls, no timeout, no exit-143), [ PASSED ] 8656, [ FAILED ] 1, then make: *** [Makefile:4713: test] Error 1 and Bash exited with code '2'. There is no AddressSanitizer report anywhere in the retrievable log, so the job did not die from a sanitizer error — it was a plain gtest assertion inside the atomic-fetch-add test that registers an atomic key on UCS_MEMORY_TYPE_RDMA (mlx5 device memory / MEMIC) and drives fadd64 over dc_mlx5. The Azure step log is truncated to the last ~500 lines, so the assertion text itself scrolled out (the test ran ~40 minutes before the summary); only the failure summary line is recoverable. The failing area is disjoint from this PR's content: the branch commits are 8c870e22 UCT/GTEST: Hardening of UD stale private EP tests, 183a66db UCT: Keep-alive for private UD EPs, 2d550625 UCT/UD/GTEST: Added unit test reproducing the failure, plus the merge 63317790 — all UD-only, touching no DC/mlx5 atomic or device-memory code. This fixture has a documented history of environment flakiness (device-memory/MEMIC allocation), and the master code it exercises was recently changed by 73a6a5c9 UCT/IB/MLX5: Handle relaxed-only fence ordering (#11721) and 1a08384f UCT/IB/MLX5: Fix put_sgl_zcopy completion (#11893), which came in via the merge commit — i.e. any real regression would belong to master, not to this PR.

Implicated commit: Not this PR's changes. Candidate master commits merged into the branch that touch mlx5 atomic/fence paths: 73a6a5c9 (Roie Danino, "UCT/IB/MLX5: Handle relaxed-only fence ordering", #11721) and 1a08384f (Zihao Zhao, #11893). Otherwise: environmental flake — unknown.

File: gtest fixture uct_atomic_key_reg_rdma_mem_type (AMO/device-memory test built on test/gtest/uct/test_amo.h:114 uct_amo_test::atomic_fop, validated by test/gtest/uct/test_amo.cc:66 validate_replies); the exact fixture definition file could not be resolved with the available read access.

Suggested fix:

1. Re-trigger the "AddressSanitizer roce on worker 2" job — a single-test failure with 8656 passes and zero ASAN output on an unrelated transport is the classic signature of a flaky/environment failure, not a PR regression. Do not block PR UCT/GTEST: Hardening of UD stale private EP tests #11920 on it.
2. To make this triageable next time, get the full gtest output instead of the truncated Azure step tail: run locally/CI with --gtest_filter='dc_mlx5/uct_atomic_key_reg_rdma_mem_type.fadd64*' --gtest_repeat=50 on an mlx5_1:1 RoCE host with an ASAN build, and archive gtest.log as a build artifact so the assertion text (Reply ... appeared N times; expected: M vs. a MEMIC/device-memory allocation or registration error) is preserved.
3. If it reproduces, bisect against 73a6a5c9 (relaxed-only fence ordering) since fence handling directly affects AMO reply correctness on dc_mlx5, and check the device-memory (MEMIC) allocation path for the known transient allocation failures — the same class of flakiness previously handled by adding retries (PR TEST/GTEST/UCT: Increase retries and add random sleep when MEMIC allocation fails #10475).

Related: #10475 (retries/random sleep on MEMIC allocation failure, same fixture family), #11721 (mlx5 relaxed-only fence ordering), #11920 (this PR)