[tvegas1]

What?

Backport #11542, can only be merged after #12051.

[tvegas1]

Verified on 2x Vera Rubin nodes (LS64-capable CPU, ConnectX), UCX built release and debug from this branch (this PR's 8eb46c0):

• Backport fidelity: diff-of-diffs against master 6e3cfd3ab is empty; only manual resolution is the bf_copy_mode field placement in ib_mlx5.h.
• ST64B really executes: gdb instruction counts are st64b=4262, generic=0, auto=4262.
• 219 two-node ucx_perftest cases pass (am/put/tag, short/bcopy/zcopy, 1 B to 2 KB, UCP with -V), plus gtest bf_copy 32/32.
• All 15 combinations of MMIO_MODE x BF_COPY_MODE pass, including a forced non-WC (NC_DEDICATED) UAR.
• With HWCAP3_LS64 masked: generic works, auto falls back, explicit st64b fails cleanly with UCS_ERR_UNSUPPORTED. Invalid values rejected at config parse.
• Performance neutral: mean latency delta vs generic is -0.00% over 103 paired configs; NIXLBench 6/6 within +/-1.6%.

[svc-ucx]

🤖 CI Triage Agent — UCX PR · commit 8eb46c00

TL;DR: A single UCP protocol-selection gtest, shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0, failed out of 10186 tests on the x86_64 CentOS8/CUDA11/ASAN GPU worker — this is unrelated to the PR (an AArch64-only ST64B BlueFlame change that isn't even compiled on this runner); retrigger and, if it persists, bisect against the recent v1.23.x protocol-selection commits.

Full analysis

Summary: make test in the gtest stage exited with Error 1 because exactly one test failed: shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0, where GetParam() = shm,ib,cuda_copy,rocm_copy. 10185 of 10186 tests passed; 5585 were skipped.

Root cause: Not caused by this PR. Evidence from the logs and source:

• The failure is in UCP protocol selection (test/gtest/ucp/test_ucp_proto.cc), which exercises am/rndv remote-proto selection under RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y + RNDV_SCHEME=get_zcopy. It touches no UCT/IB/MLX5 doorbell code.
• The container is centos8_cuda11_asan_... (x86_64). The PR commit 8eb46c00 ("UCT/IB/MLX5: Add AArch64 ST64B BF copy mode") only modifies src/uct/ib/mlx5/ib_mlx5.h and ib_mlx5.inl BlueFlame copy paths, which are AArch64-gated and therefore inactive here.
• There was no hang or timeout: timestamps run continuously, the largest gap is ~18 s (rc_mlx5/test_uct_peer_failure_multiple.test/1, a legitimately slow peer-failure test), and the suite reached global tear-down normally.
• The failing assertion is one of the three checks in check_am_rndv_remote_proto_config() (EXPECT_STREQ("am/rndv", ...), the op_id_flags equality, or EXPECT_EQ(0, count_rkey_configs_with_flag(..., UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION))). The build-tail log only carries the summary line, not the individual EXPECT output, so the exact predicate cannot be pinned from this log alone.
• This area of v1.23.x churned heavily in the two days before the run: 332d30cd (UCP/RKEY: keep configurations at stable addresses), b2a4d27d (initialize short thresholds from protocol selection), 01be4718 (make put/rndv and get/rndv a fallback of zcopy) all touch test_ucp_proto.cc / proto_select.c, and 9ec898d0 + [REDACTED:Hex High Entropy String] touch proto_rndv.c — including ucp_proto_rndv_ctrl_select_remote_proto(), which is precisely what sets UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION that this test asserts stays unset.

Implicated commit: Not [REDACTED:Hex High Entropy String] (Thomas Vegas, ST64B). Most likely one of the recent v1.23.x protocol-selection commits: 332d30cd, b2a4d27d, 01be4718 (Guy Ealey Morag) or 9ec898d0 (Thomas Vegas) — unconfirmed, since the truncated log omits the assertion text.

File: test/gtest/ucp/test_ucp_proto.cc:829 (test body) → test/gtest/ucp/test_ucp_proto.cc:449-460 (check_am_rndv_remote_proto_config); candidate production code src/ucp/rndv/proto_rndv.c:245-249.

Suggested fix: Retrigger the PR build — the ST64B change cannot affect this x86 UCP selection test, so this should not block PR #12052. To close it out properly: (1) fetch the full gtest log/artifact for this job to capture the actual EXPECT line and the UCS_TEST_MESSAGE protocol dump, since the whole-build tail drops it; (2) reproduce on the branch base without the PR commit via ./gtest --gtest_filter='shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance*' under ASAN with UCX_RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y UCX_RNDV_SCHEME=get_zcopy; (3) if it reproduces, bisect across 01be4718, b2a4d27d, 332d30cd, 9ec898d0 and focus on whether a UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION rkey config now leaks when RNDV_SCHEME is set explicitly, or whether am/rndv is no longer the selected threshold protocol at 1 MB after the put/rndv-as-zcopy-fallback change.

Related: PR #12052 (this build), PR #11542 (x86 original of the ST64B change); recently merged protocol-selection PRs #12043, #12035, #12036, #12044, #12034. Earlier investigations into this same test name: #11858, #11861, #11864, #11685.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 53149085-e717-4473-8ea8-471fe37f2b22 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.