[ofirfarjun7]

What?

backport #11727

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

ucs_topo_sys_device_enable_aux_path() breaks on the first matching DEV, so each GPU records only one data-direct sibling. On a node where two data-direct HCAs sit under the same PCIe switch as the GPUs, every HCA except that one sibling becomes unreachable for all GPUs, ends up with cuda_map == 0, and stops exposing GDA devices at all (so e.g. GDA_MAX_HCA_PER_GPU=2 is silently reduced to 1, and uct_gdaki_fill_cuda_tl_devices() returns UCS_ERR_NO_DEVICE for an otherwise usable HCA). Pre-patch both NICs were candidates. Is that intended, or should the filter apply only when the HCA's aux device is on a different switch than the GPU?

[svc-nvidia-pr-review]

Residual coverage gap: no test accompanies the change. The three behavior changes (inactive-port skip, ib_per_cuda floor of 1 when GPUs > HCAs, reachability filtering) are only exercised by test/gtest/uct/test_device.cc / test_ucp_device.cc (rc_gda), which require real GDA hardware and specific topologies. ucs_sys_topo_provider_push() cannot help because uct_gdaki_dev_matrix_init() needs real ibv_get_device_list() plus CUDA/NVML, so a focused regression test is not practical without refactoring — noting rather than requesting.

[svc-ucx]

🤖 CI Triage Agent — UCX PR (AddressSanitizer roce on worker 0) · commit 567faf3e

TL;DR: The "AddressSanitizer roce on worker 0" job never ran any UCX code — the Azure Pipelines agent swx-rain03-03 received a shutdown signal ~3 seconds into job startup, during container initialization, so this is an infrastructure flake, not a code failure. Re-run the pipeline and check the health of the swx-rain03 agent host.

Full analysis

Summary: Job roce on worker 0 (AddressSanitizer) was cancelled during the "Initialize containers" step after the Azure DevOps self-hosted agent received a shutdown signal; no build, test, or sanitizer step ever executed.

Root cause: Infrastructure/agent failure, not a product defect. Timeline from the fetched log:

• 10:31:46.35 — job starts on agent swx-rain03-03 (machine swx-rain03), agent version 3.246.0
• 10:31:48.63–10:31:49.65 — docker pull .../ubuntu22.04/builder:mofed-5.7-0.2.3.0 succeeds ("Image is up to date")
• 10:31:49.67 — ##[error]The agent has received a shutdown signal. This can happen when the agent service is stopped, or a manually started agent is canceled.
• 10:31:49.99 — ##[error]The Operation will be canceled. The next steps may not contain expected logs.
• 10:31:50.99 — ##[error]The operation was canceled.
• All subsequent steps: Skipping step due to condition evaluation.

Total job runtime was ~5 seconds with no gaps in the timestamps, so there was no hang and no timeout — the agent process/service was stopped out from under the job (agent restart, host maintenance/reboot, or OOM-kill of the agent service on swx-rain03). Because the job knob FailJobWhenAgentDies = true is set, the agent's death is reported as a job failure rather than a retry. Notably, the container image pull had already completed successfully, no repo checkout occurred, no configure/make ran, and no ASan output exists — so nothing from commit 567faf3 or PR #12074 (UCT/GDAKI: Include only usable IB HCAs in GDA matrix - v1.23.x) was compiled or executed. The failure is entirely independent of the topic/backport-use-usable-nics changes.

Implicated commit: unknown — no UCX commit is implicated; the failure predates any code execution (candidate commit [REDACTED:Hex High Entropy String] is exonerated by the log).

File: N/A — failure occurred in the Azure Pipelines agent ("Initialize containers" step, agent swx-rain03-03), not in repository source.

Suggested fix:

1. Simply re-run / re-queue the failed job — this is a transient agent-death flake and should pass on retry.
2. Investigate agent host swx-rain03 (agent swx-rain03-03): check journalctl -u vsts.agent* / the agent's _diag logs around 2026-10-12T10:31:49Z for a service stop, host reboot, or OOM-killer entry, and check for scheduled maintenance or an agent auto-upgrade that restarts the service mid-job. If the host is memory-pressured, ASan jobs (which have a large memory footprint) may be triggering OOM kills that take the agent down — consider reducing job concurrency on that machine.
3. If agent-death cancellations are recurring across PRs, consider adding a pipeline-level retry for the job (retryCountOnTaskFailure) and/or setting FAIL_JOB_WHEN_AGENT_DIES=false on these agents so an agent restart yields a retry instead of a red build, to avoid misattributing infra flakes to PR authors.

Related: PR #12074 (the PR under test — not the cause); no existing UCX issue tracks this agent-shutdown signature. Issue search for the error string returned only unrelated PRs (#11932, #12033, #11903, #11870, #11923).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id cadf6737-7a7d-403b-b301-676473118c75 in the triage console for the audit trail.