[Alexey-Rivkin]

What?

Move the distro build matrix from the Azure Build stage to Jenkins, and show failure logs on GitHub.

Why?

The Jenkins build job covered less than Azure, external contributors cannot open Jenkins logs, and running the same builds on both CIs doubles the cost of every PR.

How?

• build_matrix.yaml: the Azure rows and build modes.
• Jenkinsfile.github: on failure, upload the failed rows' logs with the Blossom lib's uploadParallelLogs, with internal hostnames hidden.
• buildlib/pr/main.yml: drop the Build stage; build_job.yml had no other user.

Tested: ucx-build-oss #46, 42/42 rows (internal NVIDIA link). Log upload on failure: ucx-test-gpu-oss #66 and ucx-test-dl-gpu-oss #60, failed rows only, hostnames hidden; the 7 green jobs uploaded nothing.

[Alexey-Rivkin]

/build

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

.ci/pipeline/build_matrix.yaml

Non-blocking: step timeout went 60 → 180 min with a pipeline cap of 240 (Azure allows 340 per leg); worth confirming the long rows fit within this.

[Alexey-Rivkin]

Non-blocking: step timeout went 60 → 180 min with a pipeline cap of 240 (Azure allows 340 per leg); worth confirming the long rows fit within this.

They fit: in ucx-build-oss #39 (internal NVIDIA link) the whole matrix took 155 min and the slowest row 152 min, inside the 180 min step timeout.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-basic-compile-oss · commit 677c88a0

TL;DR: The ucx-basic-compile-oss Jenkins job aborted immediately because the matrix config it points at, .ci/pipeline/basic_compile_matrix.yaml, does not exist in this branch (it only exists in the still-open PR #12166); either land #12166 first or stop triggering that job for branches without the file.

Full analysis

Summary: Build #8 failed during pipeline configuration with hudson.AbortException: config failed with msg: No conf_file found by .ci/pipeline/basic_compile_matrix.yaml — no compile ever ran.

Root cause: This is a CI-infrastructure mismatch, not a code/compile failure. Evidence from the console log:

1. The only stage that ran, Checkout source code, succeeded (60.4 s) and checked out merge commit [REDACTED:Hex High Entropy String] ("Merge 677c88a… into [REDACTED:Hex High Entropy String]…").
2. Immediately after the stash, matrix.main() (ci-demo library, 1f1d13d9) aborted: ERROR: config failed with msg: No conf_file found by .ci/pipeline/basic_compile_matrix.yaml.
3. That path does not exist in the repo — search_git_history on .ci/pipeline/basic_compile_matrix.yaml returns no commits ever. The matrix file that actually exists is .ci/pipeline/build_matrix.yaml (job name ucx-build, last touched by a517a209, Alexey Rivkin).
4. .ci/jenkins/Jenkinsfile.dispatcher:42-51 on this branch dispatches ucx-codestyle-oss, ucx-static-checks-oss, ucx-coverity-oss, ucx-build-oss, ucx-build-static-oss, ucx-test-oss, ucx-test-gpu-oss, ucx-test-dl-gpu-oss — ucx-basic-compile-oss is not in the list. Yet the build log shows it was "Started by upstream project UCX-OSS/ucx-dispatcher-oss build number 31".

So the ucx-basic-compile-oss job (and the dispatcher entry for it) was created on the Jenkins controller ahead of its repo-side config. That config comes from the open, unmerged PR #12166 "CI: Gate Jenkins jobs on a basic compile". PR #12198 (ci-jenkins-build-logs) is based on master [REDACTED:Hex High Entropy String], which has neither the YAML nor the dispatcher entry, so the job fails at config time for every PR that doesn't carry #12166.

Secondary defect (real, in the PR under test): the new failure-log uploader added by 677c88a0 is blocked by Jenkins script security:

Scripts not permitted to use new io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor org.jenkinsci.plugins.workflow.job.WorkflowRun
WARNING: failed to upload the failure log: ...RejectedAccessException

This is the new PipelineNodeGraphVisitor(run) call at .ci/jenkins/Jenkinsfile.github:27. Even once the config issue is fixed, this feature will silently no-op on every failure until the signature is approved. It also means no excerpt artifact was produced for this build.

Implicated commit: a517a209 / Jenkins controller job+dispatcher config added for PR #12166 (Alexey Rivkin) — the repo-side basic_compile_matrix.yaml is missing. Secondary: 677c88a0 (Alexey Rivkin, "CI: Upload failed rows' log excerpts to GitHub") for the sandbox rejection.

File: .ci/pipeline/basic_compile_matrix.yaml (missing; existing analogue is .ci/pipeline/build_matrix.yaml:5); .ci/jenkins/Jenkinsfile.github:27

Suggested fix:

1. Unblock CI now: disable the UCX-OSS/ucx-basic-compile-oss job on the controller (and remove it from the ucx-dispatcher-oss fan-out) until PR Jenkins CI: Gate Jenkins jobs on a basic compile #12166 merges. Alternatively, repoint the job's conf_file parameter to the existing .ci/pipeline/build_matrix.yaml.
2. Make this class of failure non-fatal and self-healing: have the dispatcher/Jenkinsfile.github fileExists(conf_file) and skip the job with a NOT_BUILT/neutral result instead of error, so controller-side job additions can't break PRs that predate the config.
3. Merge PR Jenkins CI: Gate Jenkins jobs on a basic compile #12166 (which adds .ci/pipeline/basic_compile_matrix.yaml and the dispatcher entry) before enabling the job, and rebase PR CI: Move the Build stage from Azure to Jenkins #12198 on it.
4. For the secondary defect, have a Jenkins admin approve the new PipelineNodeGraphVisitor(WorkflowRun) signature in Manage Jenkins → In-process Script Approval, or move failureExcerpt() into the trusted ci-demo/blossom-github-lib shared library (which runs outside the sandbox) rather than the in-repo Jenkinsfile.

Related: #12166 (adds the missing basic_compile_matrix.yaml), #12198 (PR under test), #12094 (Jenkins integration testing)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 295fc4e7-d9b4-4b56-9ffa-4231eb11881c in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-dispatcher-oss · commit 677c88a0

TL;DR: The dispatcher aborted because its gating child job ucx-basic-compile-oss #8 failed instantly with No conf_file found by .ci/pipeline/basic_compile_matrix.yaml — that matrix YAML only exists in the unmerged PR #12166, not on the branch under test, so the new gating job must not be enabled on the Jenkins controller until #12166 is merged.

Full analysis

Summary: UCX-OSS/ucx-dispatcher-oss #31 failed with hudson.AbortException: Basic compile did not pass, skipping the other jobs, caused by downstream UCX-OSS/ucx-basic-compile-oss #8 failing in config load (No conf_file found by .ci/pipeline/basic_compile_matrix.yaml).

Root cause: CI configuration/merge-order mismatch, not a product-code bug:

• The dispatcher Jenkinsfile actually executed by build UCS/UCT: Add class macro to generate new/delete function names. #31 was loaded from refs/remotes/origin/pr/12094/head (log: git rev-parse refs/remotes/origin/pr/12094/head^{commit} → 61d70cca). That version fans out to ucx-codestyle-oss + ucx-basic-compile-oss and hard-aborts the whole pipeline if basic compile fails.
• The ucx-basic-compile-oss job checked out the code under test (a63884cf = "Merge 677c88a… into [REDACTED:Hex High Entropy String]…") and then matrix.main() looked for .ci/pipeline/basic_compile_matrix.yaml. That file does not exist on this revision — read_source_file returns "not found" and search_git_history reports no commits ever touching it. The repo only has .ci/pipeline/build_matrix.yaml (job: ucx-build), and the in-tree .ci/jenkins/Jenkinsfile.dispatcher still lists ucx-build-oss, with no ucx-basic-compile-oss entry.
• basic_compile_matrix.yaml is introduced by the still-open PR Jenkins CI: Gate Jenkins jobs on a basic compile #12166 "CI: Gate Jenkins jobs on a basic compile". The Jenkins controller was already pointed at the new dispatcher/gating job before Jenkins CI: Gate Jenkins jobs on a basic compile #12166 merged, so every unrelated PR build (here CI: Move the Build stage from Azure to Jenkins #12198) fails at config-load time. This is a 100% deterministic infra failure, unrelated to the commit's content — codestyle passed, nothing was compiled.

Secondary defect visible in the same log, introduced by this PR's commit: uploadFailureExcerpt was rejected by Jenkins script security — Scripts not permitted to use new io.jenkins.blueocean.rest.impl.pipeline.PipelineNodeGraphVisitor → WARNING: failed to upload the failure log. The new failureExcerpt() helper (.ci/jenkins/Jenkinsfile.github:15,27) uses Blue Ocean internal APIs that are not sandbox-approved, so the failure-log upload feature will never work as written.

Implicated commit: a517a209 / 677c88a0 (Alexey Rivkin, "CI: Match Azure rows in the Jenkins build job" / "CI: Upload failed rows' log excerpts to GitHub") for the Jenkinsfile.github sandbox issue; the build-breaking config comes from the unmerged PR #12166 being activated on the Jenkins controller (dispatcher script pinned to pr/12094/head, 61d70cca).

File: .ci/pipeline/basic_compile_matrix.yaml (missing); .ci/jenkins/Jenkinsfile.dispatcher:42-51; .ci/jenkins/Jenkinsfile.github:15,27

Suggested fix:

1. Immediate unblock — either disable/remove ucx-basic-compile-oss from the dispatcher fan-out and revert the dispatcher job's "Pipeline script from SCM" ref from pr/12094/head back to master, or merge PR Jenkins CI: Gate Jenkins jobs on a basic compile #12166 so .ci/pipeline/basic_compile_matrix.yaml and the matching dispatcher entry land together. Never point the controller at a dispatcher revision that references YAML files absent from the branches being tested.
2. Make this failure mode non-fatal/diagnosable: have the gating job skip (and the dispatcher not abort) when the configured conf_file is missing from the checked-out revision, instead of failing the whole PR pipeline.
3. For the log-upload feature: replace the PipelineNodeGraphVisitor / BlueRun / LogAction usage in failureExcerpt() with sandbox-safe alternatives (e.g. archive the per-row log as a build artifact from inside the matrix step, or read currentBuild.rawBuild.getLog() via an approved helper), or get the signatures approved in Jenkins' In-process Script Approval. Also consider marking the helper's failure visibly rather than only echo-ing a warning.

Related: #12166 (adds the missing basic_compile_matrix.yaml), PR #12198 (commit under test), PR #12094 (source of the dispatcher script used by build #31)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id cfe647e0-a201-412e-b158-1669b5b7cc1b in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-basic-compile-oss · commit 677c88a0

TL;DR: The build never compiled anything — the ci-demo Matrix library aborted immediately because it looked for .ci/pipeline/basic_compile_matrix.yaml, which no longer exists after commit a517a209 renamed it to build_matrix.yaml (job ucx-build); fix the job-name↔file-name mismatch by renaming the Jenkins job to ucx-build-oss or pinning conf_file explicitly.

Full analysis

Summary: UCX-OSS/ucx-basic-compile-oss #9 failed with hudson.AbortException: config failed with msg: No conf_file found by .ci/pipeline/basic_compile_matrix.yaml right after the "Checkout source code" stage.

Root cause: Pure CI configuration mismatch, not a code or compile failure. The only stage that ran was Checkout source code (success, 30s); the pipeline then did fileExists / findFiles and raised error. .ci/jenkins/Jenkinsfile.github:99 calls matrix.main() with no arguments, so the ci-demo com.mellanox.cicd.Matrix library derives the matrix config path from the Jenkins job name: ucx-basic-compile-oss → .ci/pipeline/basic_compile_matrix.yaml.

That file does not exist in the tree at the tested revision (read_source_file returns "not found"). The matrix config for this pipeline is now .ci/pipeline/build_matrix.yaml, whose header declares job: ucx-build (line 5). The rename came from a517a209 "CI: Match Azure rows in the Jenkins build job" (Alexey Rivkin, 2026-10-29), the most recent commit touching .ci/pipeline before 677c88a0 under test. That commit renamed the file and the logical job name but the Jenkins job object in the UCX-OSS folder is still called ucx-basic-compile-oss, so the derived lookup path no longer resolves.

Note this failure is independent of the PR's own change (677c88a0, "CI: Upload failed rows' log excerpts to GitHub") — that commit only adds failureExcerpt/uploadFailureExcerpt to the Jenkinsfile. The upload path did run (Response Code :: 201 / 204), so the new logging code works; it just reported a config error. Every build of this job on this branch will fail identically until the name mismatch is resolved.

Implicated commit: a517a209 — Alexey Rivkin, "CI: Match Azure rows in the Jenkins build job" (renamed basic_compile_matrix.yaml → build_matrix.yaml, job: ucx-basic-compile → ucx-build)

File: .ci/pipeline/build_matrix.yaml:5 (declares job: ucx-build); lookup triggered by .ci/jenkins/Jenkinsfile.github:99

Suggested fix: Pick one of these, in order of preference:

1. Rename the Jenkins job (cleanest, matches the committed config): rename the multibranch/pipeline job UCX-OSS/ucx-basic-compile-oss to UCX-OSS/ucx-build-oss so the derived path becomes .ci/pipeline/build_matrix.yaml and agrees with job: ucx-build. Also update the GitHub branch-protection required status check, since env.JOB_BASE_NAME is used as the commit-status context in Jenkinsfile.github:94,111 — the old ucx-basic-compile-oss check will otherwise hang as pending forever.
2. Make the Jenkinsfile independent of the job name — pass the config path explicitly instead of relying on name derivation, e.g. matrix.main(conf_file: '.ci/pipeline/build_matrix.yaml') (or set the conf_file job parameter in the job definition). This makes future matrix-file renames safe and is worth doing regardless of option 1.
3. Short-term unblock without touching Jenkins: add .ci/pipeline/basic_compile_matrix.yaml as a symlink/copy of build_matrix.yaml, but keep job: consistent — this reintroduces the duplication a517a209 removed, so treat it as temporary.

Separately, consider having the Matrix wrapper fail with a clearer message that lists the .ci/pipeline/*_matrix.yaml files actually present, since the current abort gives no hint that the job name is what drives the path.

Related: PR #12168 "CI: Match Azure rows in the Jenkins static build job" (contains the renaming commit a517a209); current PR #12198 (677c88a0, failure-excerpt upload) is blocked by this but is not the cause; PR #12094 "CI: Jenkins integration testing (do not merge)" is adjacent CI work.

[Alexey-Rivkin]

/build

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

can we split the Jenkinsfile failure-excerpt change from the build matrix change? they are unrelated and carry very different risk.

Residual coverage gap: no way to verify here that the long rows (make distcheck twice, RPM/DEB build, cuda/gdrcopy variants, now with /hpc/local mounted so the CUDA module actually loads) fit the new 180 min step budget — the existing timeout comment already asks for this.

Residual coverage gap: the /hpc/local hostPath mount is a new dependency for the Blossom nodes in this job; build_static_matrix.yaml sets the precedent, but the gcc-15.2.0, ICC/PGI, armclang and CUDA module paths all degrade to silent skips if the mount is empty, so a log check is the only validation.

[Alexey-Rivkin]

can we split the Jenkinsfile failure-excerpt change from the build matrix change?

I'd keep them together: the full matrix is what makes the failure excerpt necessary, since external contributors can't open Jenkins logs, and both were tested together in ucx-build-oss #41 (internal NVIDIA link).

no way to verify here that the long rows ... fit the new 180 min step budget

The whole build took 93 minutes in ucx-build-oss #41 (internal NVIDIA link), so every row fits.

the gcc-15.2.0, ICC/PGI, armclang and CUDA module paths all degrade to silent skips if the mount is empty

The same run loads them from /hpc/local: it builds with gcc (GCC) 15.2.0, icc, icx and PGI.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[Alexey-Rivkin]

/build

[svc-nvidia-pr-review]

Residual coverage gap: rocky89-icc-pgi uses the rocky8.9 image, while the Azure icc_pgi lane runs on centos7. build_icc_check only warns when icc -v fails, so this row stays green even if ICC 19.1.1 cannot run on the newer distro — the log is the only way to tell whether anything was actually built. (Related to, but distinct from, the already-reported /hpc/local silent-skip gap.)

Residual coverage gap: No secrets are introduced by the diff; the new upload path masks internal *.nvidia.com/*.mellanox.com/*.mlnx hostnames but nothing else (e.g. internal IPs), so the masking is only as good as githubHelper.hideSensitiveData defaults, which are not visible in this repo.

[Alexey-Rivkin]

build_icc_check only warns when icc -v fails, so this row stays green even if ICC 19.1.1 cannot run on the newer distro

It runs there: in ucx-build-oss #41 (internal NVIDIA link) the row prints icc version 19.1.1.217 (gcc version 8.5.0 compatibility), then ==== Build with Intel compiler icc ====, the same for icx and PGI, and no "Not building with Intel compiler" warning.

the masking is only as good as githubHelper.hideSensitiveData defaults, which are not visible in this repo

Its defaults (getDefaultRegEx in blossom-github-lib) mask IPv4 addresses, email addresses and the pod template. The extra patterns here add the internal hostnames on top, and credential values are masked by the library as well.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-gpu-oss · commit f226351b

TL;DR: The build failed in all four parallel "Allocate Environment" stages because salloc could not get a node in the Slurm funk partition within the 600 s --immediate window — a cluster-capacity/infrastructure issue, not a code regression; fix by adding retry/backoff (or a longer wait) around the allocation and treating allocation timeouts as a non-code failure.

Full analysis

Summary: UCX-OSS/ucx-test-gpu-oss #64 aborted during the parallel Allocate Environment stages (w0–w3); every salloc returned error: Unable to allocate resources: Connection timed out, so no UCX build or test ever ran.

Root cause: Each of the 4 matrix rows ran .ci/scripts/run_slurm_allocation.sh, which issues:
salloc -N 1 -p funk --job-name=ucx-ci-w<N>-64 --immediate=600 --time=02:45:00 --no-shell
scctl login and client checks all succeeded ("login successful", "client exists"), and Slurm accepted the requests — jobs 117121/117122/117123/117124 were created and reported queued and waiting for resources at 12:05. Exactly ~600 s later (12:15:09–12:15:27) all four hit the --immediate=600 deadline and salloc exited non-zero with "Unable to allocate resources: Connection timed out".

This is not a hang: the 10-minute gap is precisely the configured SLURM_IMMEDIATE_TIMEOUT, i.e. salloc deliberately blocking while queued, then giving up. The funk partition simply had no free node for 10 minutes (plausibly aggravated by this job itself requesting 4 separate 1-node allocations concurrently). The script has no retry: scctl --raw-errors client connect -- "${SLURM_ALLOCATION_CMD}" runs once under set -e, so a transient queue wait becomes a hard pipeline FAILURE. The pipeline_stop stage then logged "WARNING: No job ID files found in /mnt/pvc/", confirming no allocation was ever granted.

The commit under test, f226351 ("CI: Upload the console tail when no matrix row failed", Alexey Rivkin), only touches CI log-upload logic and cannot cause an salloc queue timeout — the failure is unrelated to PR #12198.

Implicated commit: No code commit caused this (infrastructure). The no-retry allocation logic comes from ffc2138 "CI: Add GPU SLURM test pipeline for scctl cluster" (Iaroslav Sydoruk).

File: .ci/scripts/run_slurm_allocation.sh:163 (single-shot scctl ... client connect -- "${SLURM_ALLOCATION_CMD}"), with the timeout defined at .ci/scripts/run_slurm_allocation.sh:115 and --immediate added at line 141.

Suggested fix:

1. Short term: re-run build UCS: Add infrastructe for asyncronous events delivery. #64 — the funk partition was saturated; nothing in the PR needs changing.
2. Make allocation resilient in run_slurm_allocation.sh: wrap the client connect allocation in a retry loop with backoff (e.g. 3 attempts, 2–5 min apart) and only fail after all attempts, e.g.

   for i in $(seq 1 "${SLURM_ALLOC_RETRIES:-3}"); do
       if scctl --raw-errors client connect -- "${SLURM_ALLOCATION_CMD}"; then break; fi
       echo "WARN: allocation attempt ${i} timed out, retrying..."
       sleep "${SLURM_ALLOC_RETRY_DELAY:-120}"
   done

   Also scancel the still-pending job before retrying so queued zombies (117121–117124) don't accumulate.
3. Raise SLURM_IMMEDIATE_TIMEOUT (600 s is short for a shared GPU partition) or drop --immediate in favour of an explicit outer wait, so queue latency doesn't equal failure.
4. Consider allocating one multi-node job (-N 4) instead of four independent 1-node allocations to avoid self-contention, and have the pipeline report allocation timeouts as ABORTED/UNSTABLE rather than FAILURE so infra capacity doesn't look like a code regression on PRs.

Related: PR #12198 (branch under test, unaffected); PR #11732 "CI: use slurmCI module"; PR #12094 "CI: Jenkins integration testing (do not merge)"

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-dl-gpu-oss · commit f226351b

TL;DR: The build failed because all four parallel Slurm Allocate stages gave up after waiting the full --immediate=3600 (1 h) for a node on the gb200nvl72_ci partition ("Unable to allocate resources: Connection timed out") — a cluster-capacity/infra problem, not a UCX code regression; re-run the job and/or stop requesting whole exclusive GB200 nodes for 4 concurrent workers.

Full analysis

Summary: All 4 Allocate stages (workers w0–w3) of UCX-OSS/ucx-test-dl-gpu-oss #58 failed after ~60 min with salloc: error: Unable to allocate resources: Connection timed out, aborting the parallel matrix with script returned exit code 1.

Root cause: Pure resource starvation, not a hang in UCX code:

• Each worker stage runs salloc -N 1 -p gb200nvl72_ci --job-name=ucx-ci-dl-58-wN --immediate=3600 --time=03:20:00 --no-shell --account=oberon-gb-ci. The log shows salloc: Pending job allocation 3003467/3003604/3003605/3003613, job ... queued and waiting for resources, then the --immediate=3600 deadline expiring exactly 1 h later (e.g. submit 11:48:00 → error at 12:48:09). The ~60 min "gap" is Slurm legitimately queueing, i.e. the partition never freed a node — nothing in the test ran at all, so no UCX test hung.
• Two aggravating factors visible in the log: (a) salloc: lua: Setting job to exclusive mode as no gpu count was specified — no --gpus/--gres is passed, so every worker demands a whole exclusive GB200 node, and 4 such jobs are requested simultaneously from the same account; (b) the Jenkins side was also saturated — the build_helper_dl pod was rejected many times with exceeded quota: resourcequota-nbu-swx-ucx, requested: cpu=4500m, used: cpu=400, limited: cpu=400, confirming the whole namespace/cluster was fully booked at that time.
• The triggering commit f226351 ("CI: Upload the console tail when no matrix row failed") only touches log-upload logic and cannot cause an allocation timeout.

Implicated commit: None for the failure itself. Relevant configuration context: d5b6b27 "CI: Run the DL cluster job on the GB200 partition (#11885)", Alexey Rivkin — which moved this job onto the contended gb200nvl72_ci partition. Trigger commit f226351 (Alexey Rivkin) is unrelated.

File: Slurm allocation parameters passed to slurm.allocation (partition:gb200nvl72_ci, nodes:1, jobTimeout:03:20:00, immediateTimeout:3600) in the ucx-test-dl-gpu-oss pipeline definition; implementation lives in the external Mellanox-lab/swx-jenkins-lib@main (slurm.allocation, commit [REDACTED:Hex High Entropy String]).

Suggested fix:

1. Retry/requeue rather than fail: this is a transient capacity failure — re-run build auto-tools detection #58. Mark Allocate failures caused by Unable to allocate resources as UNSTABLE/retryable in the pipeline instead of failing the whole matrix.
2. Stop requesting exclusive nodes: pass an explicit --gpus=N/--gres=gpu:N in extraArgs so the lua plugin doesn't escalate each job to exclusive whole-node mode; 4 exclusive GB200 nodes per build is what makes the queue unsatisfiable.
3. Reduce concurrency: serialize or lower the parallel worker count (4 → 1–2) for the DL GPU job, or make workers share one allocation instead of 4 separate sallocs.
4. Separately, raise/clean up the resourcequota-nbu-swx-ucx namespace CPU quota (currently 400 CPU, fully consumed, blocking the 4500m helper pod) with the Blossom/infra team — the build_helper_dl pod needed ~10 retries to schedule.
5. Do not just raise immediateTimeout beyond 3600 s — the jobs would still queue behind unavailable exclusive nodes and only delay the failure.

Related: PR #12198 (trigger, CI log-upload work), PR #11885 / commit d5b6b27 (moved job to GB200 partition), PR #12094 ("CI: Jenkins integration testing") — no existing issue tracking the GB200 allocation timeouts; worth opening one against the CI/infra team.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 675e642e-39da-453f-a4b4-cc8b17e577f8 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[Alexey-Rivkin]

/build

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-gpu-oss · commit affcce21

TL;DR: The gtest shard 3 of build #66 hung — rcx/test_ucp_rma_reg.get_nonblocking/1 (rc_x, ODP/REG_NONBLOCK host registration, MAX_RMA_LANES=2) issued ucp_get_nbx operations that never completed, so the worker flush never finished, data validation failed and the gtest watchdog aborted after ~12 minutes of total silence; this is an RMA/GET completion hang unrelated to PR #12198 (a CI-only branch), not a timeout that needs raising.

Full analysis

Summary: Stage "Test" (node 158) failed: gtest shard 3/4 on node funk18 hung in rcx/test_ucp_rma_reg.get_nonblocking/1 and was killed by the test watchdog (Connection timed out - abort testing, SIGABRT + core dump). Three sibling "Allocate Environment" stages separately failed with salloc: error: Unable to allocate resources: Connection timed out.

Root cause: A hang, not slowness. Timeline from the stage log:

• 16:20:32 [ RUN ] rcx/test_ucp_rma_reg.get_nonblocking/1 <rc_x,cuda_copy,rocm_copy/nb>
• 16:23:38 ucp_test.cc:310 UCX ERROR request 0x555559945a20 did not complete on time → test_ucp_memheap.cc:115: data validation failed, with progress printed only up to host->host 5x128 5x894 5x9433 5x230836 (it stalled on the next, ~3.7 MB size step)
• 16:26:45 a second request did not complete on time (cleanup/flush path)
• 16:26:45 → 16:38:37: ~12 minutes with zero output (the largest gap, and most of the test's runtime), then the watchdog fired; the backtrace shows the main thread spinning in sched_yield inside the gtest progress loop.

So the ucp_get_nbx requests (and the subsequent worker flush / cleanup flush) never completed — a lost/never-delivered GET completion on the rc_x transport. The variant /nb sets UCX_REG_NONBLOCK_MEM_TYPES=host (ODP, non-blocking registration) and test_ucp_rma forces MAX_RMA_LANES=2, i.e. the multi-rail GET zcopy path over ODP-registered memory. PR #12198 / branch ci-jenkins-build-logs only touches Jenkins build-log handling, so it cannot have caused this; it is a pre-existing bug/flake on master or a wedged HCA/QP on funk18.

The Allocate Environment failures are a separate symptom: salloc --immediate=600 on partition funk gave up after 10 min because no node was free (partly because this very job held funk18 for 2h45m while hung).

Implicated commit: unknown — not from the PR under test. Most plausible recent candidates on the RC/GET completion path, all merged shortly before this run: 757f9f9b "UCT/IB/MLX5: support rma get in purge outstanding (#11866)" (Zihao Zhao), 68cfcbc5 "UCT/IB/MXL5: uct comp marker (#11915)" (Zihao Zhao), 51224230 "UCT/IB/MLX5/RC: Implement get_sgl_zcopy (#11763)" (Michal Shalev), and [REDACTED:Hex High Entropy String] "UCP/PROTO: Repost GET and zcopy on the selected lane (#12091)" (Evgeny Leksikov).

File: test/gtest/ucp/test_ucp_memheap.cc:115 (failure point); test/gtest/ucp/test_ucp_rma.cc:880-974 (test_ucp_rma_reg, nb variant → REG_NONBLOCK_MEM_TYPES=host); suspect code in the rc_mlx5 GET zcopy / outstanding-completion path (src/uct/ib/mlx5/rc/) and src/ucp/rma.

Suggested fix:

1. Do not raise the Slurm/stage time limit — the process was hung, not slow.
2. Reproduce locally on a funk node: UCX_REG_NONBLOCK_MEM_TYPES=host UCX_MAX_RMA_LANES=2 ./gtest --gtest_filter='rcx/test_ucp_rma_reg.get_nonblocking/1' with UCX_LOG_LEVEL=debug; the run already produced a core dump, so get a backtrace of the stuck worker and dump the rc_x QP/CQ state to see whether the GET WQE was posted and its completion never consumed (an ODP page-fault stall vs. a dropped completion in the purge-outstanding/comp-marker rework).
3. Bisect across 757f9f9b, 68cfcbc5, 51224230, [REDACTED:Hex High Entropy String] — all touch exactly the RC GET-zcopy completion bookkeeping that this test exercises.
4. Check funk18's HCA health (ibv_devinfo, port counters, QP in error) to rule out a wedged device; drain the node if it is bad.
5. CI hygiene: the build archived no artifacts — add archiving of the core dump and gtest stdout for this stage so hangs are debuggable, and make the hung test fail fast instead of holding the Slurm allocation for 2h45m, which is what starved the other three shards' salloc --immediate=600.

Related: PR #12198 (build trigger, CI-only). Candidate culprits: #11866, #11915, #11763, #12091. No existing issue found for "request did not complete on time" in test_ucp_rma_reg.get_nonblocking — worth opening one.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c0098d81-c753-4f02-8e16-84001148de28 in the triage console for the audit trail.

[svc-ucx]

🤖 CI Triage Agent — UCX-OSS/ucx-test-dl-gpu-oss · commit affcce21

TL;DR: No UCX code was tested — all four DL‑GPU worker branches died in the Allocate stage: three waited the full --immediate=3600 (1 h) for an exclusive GB200 node and got salloc: error: Unable to allocate resources: Connection timed out, while the fourth (w2) crashed instantly with groovy.lang.MissingPropertyException: No such property: slurm for class: slurmCI because it referenced the shared library's slurm global before another parallel branch had finished loading it. Fix: load swx-jenkins-lib once before the parallel block, and treat the 1‑h slurm queue timeout as a capacity/infra issue (retry or reduce the number of simultaneous exclusive node requests).

Full analysis

Summary: UCX-OSS/ucx-test-dl-gpu-oss #60 failed in the Parallel/Allocate stages — no GB200 slurm allocation was ever obtained and one branch threw a Groovy MissingPropertyException.

Root cause: Two independent problems, neither in UCX source:

1. Slurm capacity (branches w0, w1, w3). Each branch ran salloc -N 1 -p gb200nvl72_ci --immediate=3600 --time=03:20:00 --no-shell --account=oberon-gb-ci. The log shows salloc: lua: Setting job to exclusive mode as no gpu count was specified, then Pending job allocation 300855x, job ... queued and waiting for resources, and after exactly 3600 s error: Unable to allocate resources: Connection timed out. The gap in the log (15:39:09 → 16:39:17, 15:39:42 → 16:39:51, 15:41:25 → 16:41:33) is the --immediate wait itself, i.e. the jobs sat in the slurm queue the whole hour — four concurrent exclusive whole‑node GB200 NVL72 requests never got scheduled. This is a cluster/partition capacity problem, not a hang in UCX.

2. Race loading the shared library (branch w2, failed in 962 ms). Stage 189 printed only Calling slurm.allocation with args=[... jobName:ucx-ci-dl-60-w2 ...] at 15:39:06.885 and then aborted; the build's final error is hudson.AbortException: parallel task failed with msg: groovy.lang.MissingPropertyException: No such property: slurm for class: slurmCI. Branch w1 was the one actually loading the library (Loading library swx-jenkins-lib@main 15:39:04 → checkout complete 15:39:08.6); w0/w2/w3 only logged Only using first definition of library swx-jenkins-lib. w2 tried to resolve the library global slurm at 15:39:06, i.e. before the library checkout finished, so the binding wasn't populated yet. This exception is what Jenkins reported as the build result, masking the real allocation timeouts.

Secondary noise (not fatal, but it delayed/serialized the branches): the k8s namespace quota was exhausted while provisioning the helper pods — repeated pods "ucx-test-dl-gpu-buildhelperdl-60-..." is forbidden: exceeded quota: resourcequota-nbu-swx-ucx, requested: cpu=4500m, used: cpu=398900m, limited: cpu=400, retried for ~30 s before a pod was created. Also pipeline_stop reported WARNING: No job ID files found in /mnt/pvc/ucx-test-dl-gpu-oss/, so the pending slurm jobs were never explicitly cancelled.

Implicated commit: CI-side only. Head commit [REDACTED:Hex High Entropy String] (Alexey Rivkin, "CI: Upload only the failed rows' logs to GitHub") is unrelated to the failure mode. Relevant earlier CI changes: d5b6b278 (Alexey Rivkin, "CI: Run the DL cluster job on the GB200 partition", #11885) which moved this job onto the contended gb200nvl72_ci partition, and the shared-library call pattern introduced via PR #11732 ("CI: use slurmCI module"). The library itself is at Mellanox-lab/swx-jenkins-lib@ea82340 ("slurm: log submit host for salloc and srun SSH sessions").

File: swx-jenkins-lib consumer in the DL pipeline — the per-branch library 'swx-jenkins-lib@main' + slurm.allocation(...) call site (stage Allocate, build #60 node IDs 184/189/229/249); the library side is vars/slurm.groovy / slurmCI in Mellanox-lab/swx-jenkins-lib (not present in the ucx repo, so no line number available).

Suggested fix:

1. Load the shared library exactly once, outside the parallel block — e.g. a top-level @Library('swx-jenkins-lib@main') _ annotation or a single library 'swx-jenkins-lib@main' before branches fan out — and/or bind the returned handle (def lib = library(...)) instead of relying on the implicit slurm global inside each branch. This removes the MissingPropertyException race deterministically.
2. Wrap slurm.allocation in a retry with backoff and make the failure message distinguish "queue timeout" from "pipeline error", so a capacity shortage doesn't surface as a Groovy exception.
3. For the allocation timeouts: either request GPUs explicitly (avoiding the lua hook's forced --exclusive whole-node mode), lower the worker count so four exclusive GB200 nodes aren't needed simultaneously, or raise the issue with the gb200nvl72_ci / oberon-gb-ci owners — do not simply raise immediateTimeout, the jobs were queued, not running.
4. Fix pipeline_stop so job IDs are written to /mnt/pvc/ucx-test-dl-gpu-oss/ before salloc blocks (currently the id file is only written on success), otherwise orphaned pending allocations are left behind.
5. Investigate the namespace CPU quota (limited: cpu=400, used: 398900m) — the pod-provisioning retries indicate the namespace is saturated by concurrent builds.

Related: #12198 (this PR), #11732 ("CI: use slurmCI module"), #11885 ("CI: Run the DL cluster job on the GB200 partition"), #12094 ("CI: Jenkins integration testing")

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id cb3f7f4d-9fd1-4fff-8227-5caa08e8db72 in the triage console for the audit trail.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

can we mount /auto/sw_tools here as well? the Azure build lanes used -v /hpc/local:/hpc/local -v /auto/sw_tools:/auto/sw_tools (and coverity_matrix.yaml mounts /auto/sw_tools/Commercial for the commercial tools/cov-* tree). az_init_modules only adds /hpc/local/etc/modulefiles to MODULEPATH, so a modulefile can load while its payload stays unreachable; if the Intel/PGI payload is not reachable build_icc_check only calls azure_log_warning, so the new rocky89-icc-pgi/compilers row stays green without building anything.

[svc-nvidia-pr-review]

do the extra keys in the docker rows (build_mode, GCC_MODULE, CONTAINER, require_ze) really reach builds.sh as env? Every other matrix in .ci/pipeline passes per-row values through matrix.axes or env (e.g. ${type} in coverity_matrix.yaml), and build_static_matrix.yaml even flattened the Azure per-container values into global env. A value that does not arrive fails silently - build_ze just prints "Not building with ZE (require_ze!=yes)" and exits 0, and build_mode now falls back to long since the build_mode: "short" env default was removed.

[svc-nvidia-pr-review]

minor: pls update the header of buildlib/dockers/ze-builder.Dockerfile (lines 7-11) - it still documents that the image is consumed by the ubuntu2404_ze container in buildlib/pr/main.yml and that main.yml's image ref must stay in sync, but after this PR the ZE lane is this row on a different registry host.

[svc-nvidia-pr-review]

minor: the container resources that only build_job.yml referenced are now dead config (rhel82, rhel90, rhel100, rhel100_aarch64, rocky89/96(+_aarch64), debian113/125/130, sles15sp7/16sp0(+_aarch64), fedora44(+_aarch64), ubuntu2604/2610(+_aarch64), azurelinux40(+_aarch64), openeuler*, uos201060a*, tencentos44*, ctyunos2507*, centos10stream, ubuntu2204_rocm_6_0_0, ubuntu2404_ze, ubuntu2404_aarch64) together with the DOCKER_OPT_ZE_GPU variable that only that container used; pls drop them with the stage.

[svc-nvidia-pr-review]

The three $(build_mode) / $(require_ze) / $(require_real_ze) Azure-literal workarounds in buildlib/tools/builds.sh:14-29 become dead after this PR: the only remaining Azure caller is the soname job at main.yml:525, which sets build_mode explicitly and never passes the ZE variables. Not raised inline since the file is not in the diff.

No way to verify here that the Jenkins helper exposes uploadParallelLogs(this, job, build, null, <regex list>) with that signature, or that its masking covers more than the three hostname patterns; blossom-github-lib is external to this repo.