[yafshar]

What?

Extract the accelerator device-selection logic shared by the CUDA, ZE and ROCm perftest allocators into ucx_perf_get_gpu_index() in libperf_int.h, and use it from all three. The ZE and ROCm allocators now honor the device id given by -a <type:dev-id>, and an out-of-range id is rejected at allocator init.

Why?

An explicit device id was silently ignored by the ZE and ROCm allocators: only the CUDA allocator read params.{send,recv}_device.device_id, so a run requested on a specific card was placed by group_index % num_gpus instead, with no diagnostic. ROCm also computed that modulo without checking the device count, which divides by zero when hipGetDeviceCount() reports none. The same selection logic was duplicated in three allocators.

How?

A single helper, used by all three allocators. -a does not select GPU memory on its own; -m selects the allocator, and the id then applies to it. For UCT with -m ze-* / -m rocm*, the perftest allocator allocates the buffers (.uct_alloc), so the id selects the device they are placed on.

Scope: with this PR the ZE/ROCm device id takes effect for UCT tests. For UCP, these allocators define no .mem_alloc, so buffers are allocated by the memory domain (UCP_MEM_MAP_ALLOCATE), not placed by the selected id. Moreover, -a switches the run into device-API mode, whose dispatcher is registered for CUDA memory types only and implements only PUT tests. A regular UCP test such as tag_bw with -a therefore fails with Invalid test case on both master and this branch, including with CUDA. Separating device selection from device-API mode will be done in a follow-up PR.

Testing (master 8cfe4b803 vs this branch, same configure flags, same commands):

• CUDA, NVIDIA L40S, CUDA_VISIBLE_DEVICES=1,2: same device-count and upper-bound checks, no device-selection behavior change. The device chosen for -a cuda:0, -a cuda:1, -a cuda:0,cuda:1 and no -a is identical between master and this branch, and UCP tag_bw -m cuda bandwidth matches. The only difference is the error text for an out-of-range id: Illegal cuda device 2 number of devices 2 becomes illegal cuda device 2, number of devices 2.
• ZE, Intel Arc Pro B60, ZE_AFFINITY_MASK=1,2, with -x rc_verbs -d mlx5_0:1 -t am_lat -D zcopy -m ze-device:
  • An out-of-range id (-a ze-device:2, -a ze-device:9) is rejected on both processes (illegal ze device N, number of devices 2); master accepts it silently.
  • On master, the buffer addresses are the same for every -a value (placement by group index only). On this branch they follow the requested id, including split send/recv ids, and are unchanged without -a. Placement was inferred from the device-specific address ranges, not queried from the driver.
  • A completed UCT benchmark was not obtained on this setup: ibv_reg_mr() fails with Bad address for ZE device pointers, because this UCT registration path supplies no DMA-BUF fd and ib_md.c therefore takes the VA path. The failure is identical on master and without -a, so it is pre-existing and unrelated to this change.
  • UCP tag_bw -m ze-device without -a completes with matching bandwidth on master and this branch.
• ROCm: not built against a real HIP SDK and not executed; no AMD hardware available.
• CI builds ZE with the null driver and builds ROCm when its SDK is available; neither path exercises perftest device selection on hardware.

Pre-existing limitation found while testing, not addressed here: UCX_PERF_CMD_PUT with -a zeroes required_flags/max_size, so that UCT path cannot pass capability checking.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

Residual coverage gap worth noting: there is no test or CI job exercising -a rocm:<id> / -a ze-device:<id>, and the rocm/ze paths need real hardware, so the new validation and error paths for those two modules are unverified by CI.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (Tests gpu on worker 0) · commit fd4baa9e

TL;DR: The gpu worker's gtest run failed on a single unrelated protocol‑selection test, shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0; the PR (perftest ZE/ROCm device id) cannot affect it, and the test is environment-sensitive because it hard-asserts that am/rndv is selected for 1 MB CUDA AM sends with RNDV_SCHEME=get_zcopy while only guarding on generic rndv support. Re‑trigger the job, and harden the test with a CUDA‑capable‑NIC guard (require_cuda_net_md()).

Full analysis

Summary: make test in the gtest stage returned error 1 with exactly one failure out of 10306 tests: shm_ib/test_ucp_proto_am_rndv.am_explicit_scheme_disables_force_provenance/0 (GetParam() = shm,ib,cuda_copy,rocm_copy), on host swx-rdmz-ucx-gpu-02 in the ubuntu2404_doca31_gpunetio container.

Root cause: The log shows a legitimate assertion failure (not a timeout/hang — the run finished tear-down normally and all other tests passed). The failing test runs with RNDV_PIPELINE_SHM_CUDA_STAGING_FORCE=y + RNDV_SCHEME=get_zcopy and calls check_am_rndv_remote_proto_config(0, 0), which in turn does EXPECT_STREQ("am/rndv", thresh->proto_config.proto->name) for a 1 MB CUDA AM send (test/gtest/ucp/test_ucp_proto.cc:438). Because RNDV_SCHEME=get_zcopy forces rndv_mode != AUTO (src/ucp/rndv/proto_rndv.inl:25-27), the rendezvous control protocol is only selectable if a remote rndv/get/zcopy candidate exists, which requires an endpoint lane whose MD can register CUDA memory. Unlike the sibling tests in the same file (test_ucp_proto_mock_rcx_*, rndv_force_cuda_*) which call require_cuda_net_md(), the am_* variants only call require_cuda_rndv_support() — so on a node/configuration where the selected shm/ib lanes cannot register CUDA memory (GPU↔NIC assignment on this DOCA/GPUNetIO GPU node), a non-rndv eager protocol is selected at 1 MB and the test fails instead of skipping. This test family is the one introduced/reworked by the "remote sys-dev namespace / provenance" work (#11685) and was already the subject of dedicated diagnosis PRs for Azure protocol-selection instability (#11858, #11861, #11864). Note: the per-assertion gtest text is no longer in the retrievable Azure log tail (only the final [ FAILED ] summary survives), so the exact EXPECT_* line is inferred from the test code, not quoted from the log.

The PR under test (#11993, "TOOLS/PERF: Honor configured device id in ZE and ROCm allocators") touches only src/tools/perf allocators and cannot influence UCP protocol selection — this is not a regression from this PR.

Implicated commit: 0311ed7 — jeffnvidia, "UCP/PROTO/RNDV: Isolate remote system device namespace (#11685)" (added the *_force_provenance tests and the UCP_RKEY_CONFIG_FLAG_PROTO_ESTIMATION path); recent topology-assignment changes on the same nodes are contributing context (fd1eb5e / [REDACTED:Hex High Entropy String], Guy Ealey Morag).

File: test/gtest/ucp/test_ucp_proto.cc:805-813 (test body) and test/gtest/ucp/test_ucp_proto.cc:438 (EXPECT_STREQ("am/rndv", ...)); config gate at src/ucp/rndv/proto_rndv.inl:20-28.

Suggested fix:

1. Re-run the gpu job for PR TOOLS/PERF: Honor configured device id in ZE and ROCm allocators #11993 — the failure is unrelated to the change set.
2. Harden the test: in test_ucp_proto_am_rndv (and the test_ucp_proto_rma_rndv *_force_provenance twins) replace require_cuda_rndv_support() with require_cuda_net_md(), or skip when select_am_rndv_remote_proto_config() does not yield am/rndv (e.g. UCS_TEST_SKIP_R("rndv get_zcopy over CUDA is not available")) instead of EXPECT_STREQ, so nodes without a CUDA-registrable NIC lane skip rather than fail. The assertion of interest (count_rkey_configs_with_flag(PROTO_ESTIMATION) == 0) can still be checked unconditionally.
3. CI improvement: publish the gtest XML/--gtest_output or the full log as a build artifact — the whole-build tail truncates the failing assertion, which made triage rely on source inspection.

Related: #11685, #11858, #11861, #11864, #11990 (topology assignment on DPU-equipped nodes), PR under test: #11993

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 63a558e9-9863-4f5d-8c0e-5fe57edde427 in the triage console for the audit trail.

[tvegas1]

-a currently also switches the run into device api mode: it adds UCP_FEATURE_DEVICE (libperf.c:958), moves thread_count into device_thread_count (perftest_params.c:866), and routes ucp dispatch through ucx_perf_mem_type_device_dispatchers[] (ucp_tests.cc:1209), which is registered for cuda only. So using -a to pick a ze/rocm device also pulls in device mode semantics that those allocators do not support.

Can we add a separate knob to enable device mode, and keep -a for device type and id only, as its help text already says? Then cuda could also select a device in a regular run, which is not possible today.

[yafshar]

Agreed, -a currently mixes device selection with device-API mode, and this is pre-existing: on master, ucx_perftest -t tag_bw -m cuda -a cuda:1 also fails with Invalid test case, so CUDA can't select a device in a regular UCP run either. Adding a separate knob changes CLI semantics that CI relies on (test_jenkins.sh uses -a cuda:0 to enable device mode), so I'd like to keep this PR limited to honoring the id in the allocators and do the knob in a follow-up PR. With this PR alone, the ZE/ROCm device id takes effect for UCT tests.

[tvegas1]

-a today is only usable for the device feature: it implies UCP_FEATURE_DEVICE (libperf.c:958), moves thread_count into device_thread_count (perftest_params.c:866) and routes ucp dispatch through ucx_perf_mem_type_device_dispatchers[] (ucp_tests.cc:1209), which is registered for cuda only. So it would make ze/rocm honor a device id only in a mode those allocators do not support, and no allocator supports placement outside that mode today.

Can we instead move the device id to -m, with the same <type>:<dev-id> syntax, and keep -a without argument to only enable the device feature? -m already selects the allocator that places the buffers, and it avoids the strange -m ze-device -a ze-device:0.

[tvegas1]

actually started #12003 that should cover this PR, if you could confirm?

[yafshar]

Yes, #12003 subsumes this PR: the shared device-index helper, ZE/ROCm honoring an explicit id, the range checks and the ROCm zero-device guard, and it also separates device mode from device selection, which answers the open -a threads and the type-mismatch question here. Since it targets master independently and moves the id from -a <type:id> to -m <allocator:id>, I'll treat it as a replacement and close this PR once #12003 is merged. I left a few notes on #12003; the main one is preserving the GPU-0 pinning in the CUDA device CI job.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

-a <type>:<id> today is also the switch that turns the test into device-initiated mode: adjust_test_params() folds -T into device_thread_count and forces thread_count=1, and ucp_perf_test_dispatch() then only uses ucx_perf_mem_type_device_dispatchers[mem_type], which is registered for cuda only. So -a rocm:1 / -a ze-device:1 on a UCP test ends in "Invalid test case". How is device-id selection meant to work for rocm/ze — separate option, or is this UCT-only for now?

Residual coverage gap: no CI job exercises -a rocm:<id> / -a ze-device:<id>, and both need real hardware.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

minor: should we also check that the -a device type matches the allocator? Only device_id is read here (without looking at send_device.mem_type/recv_device.mem_type), so a device id given for one accelerator type is applied to whichever allocator happens to run init() — e.g. -m rocm -a cuda:1 silently selects rocm device 1 instead of reporting a mismatch. Low impact (mostly a mistyped-argument case) and partly pre-existing for cuda, but this PR is the point where the code becomes shared across three backends. This overlaps the still-open thread about -a semantics for rocm/ze, so it may be simplest to answer it there.

[yafshar]

Closing: superseded by #12003 (merged), which covers the shared device-index helper, ZE/ROCm honoring an explicit device id, the range checks and the ROCm zero-device guard, and also separates device mode from device selection.

Thanks @tvegas1