[tvegas1]

What?

Correct Vera Rubin CUDA-copy and GPU–HCA bandwidth estimates.

Internal: 6746767

Why?

#11749 increased CUDA-copy bandwidth, making direct zcopy appear slower than host staging. Correct per-lane GPU–HCA bandwidth lets UCX aggregate all four lanes accurately.

How?

Case below where both initiator and target memory are accessible by HCA, zcopy protocols being disabled.

Before:

1159079..inf  pipeline cuda_copy, write to remote, frag host  4 × 25% rc_mlx5

After:

10947..inf  zero-copy fenced write to remote  4 × 25% rc_mlx5

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Coverage note: there is no CI coverage for this change — config/ucx.conf is only installed (config/Makefile.am:8) and the gtest config tests use their own test/gtest/ucs/ucx.conf. The values are only exercised on real Vera hardware, so please confirm they were validated on a Vera Rubin system.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 1) · commit 88aad1e4

TL;DR: The build failed on a single gtest case — rc_gda/test_ucp_device_xfer.put/14 <rc,rc_gda,cuda_copy,rocm_copy/host/warp/multi_channel> — i.e. a GDA (device‑API) multi‑channel PUT into host remote memory; everything else (10243 tests) passed, and the failure is in GDA lane/channel selection code that changed in master the day before this run, not in the PR's config-only change. Re-run to confirm flakiness, and if it reproduces, capture the gtest assertion and check how many device lanes (num_lanes) are selected for a host-memory remote list.

Full analysis

Summary: make test in build-test/test/gtest returned Error 1 because exactly one test failed: rc_gda/test_ucp_device_xfer.put/14 (params rc,rc_gda,cuda_copy,rocm_copy/host/warp/multi_channel); the job then exited with code 2.

Root cause: From the log, the run was healthy (no hang — continuous per-test timestamps up to teardown, 10264 tests, 20 skips, 1 failure) and the only failing case is the GDA device-PUT variant that is unique in three ways: it sets UCX_RC_GDA_NUM_CHANNELS=32 and passes num_channels=32 (test/gtest/ucp/test_ucp_device.cc:584-586, 609-611), it uses host memory as the remote/destination type, and it runs at warp level (128 threads → 4 warps, channel_id = warp_index % 32 → 0..3, test/gtest/ucp/cuda/test_kernels.cu:29-32). Only in this multi_channel mode is channel_id != 0, and UCP_DEVICE_GET_LANE maps lane = channel_id % handle->num_lanes (src/ucp/api/device/ucp_device_impl.h:108-110) — so this is the only variant that exercises secondary device lanes / non-zero UCT channels. The sibling variants (host/thread/multi_channel, cuda/*/multi_channel) passed, so the defect is confined to the multi-lane/multi-channel GDA put path for host remote memory. The exact assertion text is not in the retrievable log tail (Azure only returns the last ~500 lines of the task log), so the precise assertion (kernel status vs. dst_pattern_check) cannot be quoted. Note that master merged GDA changes hours before this build: c82c8fcd "UCT/GDAKI: Restore 'Check for HCA reachability like rc_mlx5 (#11781)'" (#11945, 2026-09-14, Artemy Kovalyov — previously merged, reverted twice, then restored) and a1154927 "UCT/IB/MLX5/GDAKI: Try VMM Fabric Allocation" (#11795). Both change which HCAs become GDA device lanes, i.e. exactly the num_lanes value that the failing variant depends on. The PR under test only raises the UCX_DISTANCE_BW default, which also feeds GPU↔NIC distance-based lane selection, so it cannot be fully excluded — if it enlarges the set of qualifying NICs, num_lanes grows and lanes >0 (possibly with a host-memory