Repository navigation
CI: Replace Blue Ocean with pipeline-overview URL - #11796
Conversation
|
/build |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage Root cause: The CI authors check scans the commit range Implicated commit: [REDACTED:Hex High Entropy String] — Noam Tsemah ntsemah@nvidia.com (branch HPCINFRA-4739, PR #11796) File: AUTHORS:90-91 Suggested fix: Add the author to Nir Wolfer <nwolfer@nvidia.com>
+Noam Tsemah <ntsemah@nvidia.com>
Ofir Farjon <ofarjon@nvidia.com>Include this change in PR #11796 (either as part of commit Related: PR #11796 (this PR). No existing issue found for this failure; it is a per-PR onboarding step rather than a recurring CI bug.
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Evidence from stage 80/37 log:
So the classic Environment Modules 3.2.x Implicated commit: unknown — not a code regression from File: Suggested fix:
Security note: the stage log dumps the full environment via Related: none found (searched issues/PRs for |
|
🤖 CI Triage Agent — TL;DR: All Coverity stages failed because Full analysisSummary: Root cause: Environment/agent issue, not a code regression in the commit under test. The stage log shows:
The (old, 3.x) Environment Modules on this Jenkins node aborts the directory walk of Implicated commit: unknown (no UCX source commit implicated; the last functional change in this area is 5c4ddb1 "BUILD/CI: Add ZE null-driver build and smoke check (#11569)", Yaser Afshar, which reworked File: buildlib/az-helpers.sh:114-136 ( Suggested fix:
until module avail -t "$module" 2>&1 | grep -q "^${module}\$" ||
module load "$module" 2>/dev/null; door test existence explicitly: Related: none found (searched issues/PRs for this signature) |
|
🤖 CI Triage Agent — TL;DR: No UCX code failure — all four parallel GPU test workers failed in the "Allocate Environment" stage because their Slurm Full analysisSummary: Root cause: Each worker branch runs
The ~10-minute gap in the log is not a hang in the build: it is Note also that the build commit 4ffd98a ("CI: Replace Blue Ocean with pipeline-overview URL", Noam Tsemah) only changes a URL string and cannot cause this. Secondary observation: the Implicated commit: unknown (infrastructure/capacity). Script origin: File: Suggested fix:
Related: PR #11796 (branch HPCINFRA-4739) — no existing issue found for this Slurm allocation timeout; consider opening one to track allocation retry handling. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Build #4, stage 160 "Checkout & test" (ABORTED after 8,410 s) — the Root cause: Timestamp analysis of the stage log shows continuous activity through all
That single gap is ~92 % of the stage runtime, so this is a hang, not insufficient time. Neither process emitted anything after the client launched — the server never printed The reason this consumed the whole job instead of failing is Implicated commit: [REDACTED:Hex High Entropy String] (PR #11796, branch HPCINFRA-4739) is the change under test and the prime suspect — File: Suggested fix:
Related: PR #11796 (the change under test); no existing issue found for a stream client‑server hang — the only near match from a repo search was unrelated (#4353, client/server connection with atomics).
|
|
/build |
|
🤖 CI Triage Agent — TL;DR: Both Coverity stages failed because Full analysisSummary: Root cause: Because of the dangling So this is not an autofs/timing issue (retries and Implicated commit: unknown — not a source regression; the CI agents' File: Suggested fix:
function az_module_load() {
module=$1
retries=5
until module show "$module" >/dev/null 2>&1; do
if [ $retries -gt 1 ]; then
echo "Module $module not found, retrying..."
ls /hpc/local >/dev/null 2>&1
sleep 1
else
echo "MODULEPATH='${MODULEPATH}'"
module avail || true
ls -l /hpc/local/etc/modulefiles/"$module" || true
azure_log_warning "Module $module cannot be loaded"
return 1
fi
((retries--))
done
module load "$module"
}
Related: PR #11796 (#11796) — the triggering PR, unrelated to the failure; no existing issue found for the |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Root cause: Not a UCX code/build failure and not a timeout. The log shows A stale/dangling Implicated commit: unknown (no UCX commit implicated; File: Suggested fix:
for d in ${MODULEPATH//:/ }; do
if [ -f "$d/$module" ]; then
module load "$module" && return 0
fi
doneand/or filter modulecmd's Security note: the stage log dumps the full environment ( Related: PR #11796 (trigger only, unrelated); no existing issue found for this module-load failure.
|
|
🤖 CI Triage Agent — TL;DR: All four gtest workers hung (not timed out on Jenkins' clock — genuinely stuck) in Full analysisSummary: Stage Root cause: UCP RMA put/get requests hang indefinitely when the local buffer is registered non-blocking. Per-worker failing cases from the logs:
All four are Implicated commit: Not the PR head (0697237 / PR #11796 is "CI: Replace Blue Ocean with pipeline-overview URL" — no library code). Most plausible code suspects touching IB memory-key/registration behaviour: File: Suggested fix:
Related: PR #11796 (trigger, CI-only) #11796; PR #11808 #11808 (recent RMA lane-selection change, secondary suspect). No existing issue found for this hang — worth opening one with the four failing test names above.
|
|
🤖 CI Triage Agent — TL;DR: The Coverity stages fail because Full analysisSummary: Both Root cause: In the Coverity stage log, Environment Modules reports:
Because Implicated commit: unknown — not caused by File: Suggested fix:
function az_module_load() {
module=$1
retries=5
until module load "$module" 2>/dev/null; do
if [ $retries -gt 1 ]; then
echo "Module $module not loadable, retrying..."
ls /hpc/local > /dev/null 2>&1 # refresh automount
sleep 1
else
echo "MODULEPATH='${MODULEPATH}'"
module avail || true
ls -l /hpc/local/etc/modulefiles/"$module" || true
azure_log_warning "Module $module cannot be loaded"
return 1
fi
((retries--))
done
}
Related: PR #11796 (the triggering PR — unrelated to the failure); same failure reproduced in
|
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "commit title" codestyle job failed because PR #11796 contains a commit titled Full analysisSummary: Azure Pipelines job Root cause: This is a genuine lint failure, not infrastructure flakiness.
The step ran only ~30 ms with no gaps, so there is no timeout/hang aspect. Secondary defect: the step script uses Implicated commit: [REDACTED:Hex High Entropy String] (PR #11796 head, the commit titled File: buildlib/tools/codestyle.sh:14 (regex) — triggered via buildlib/pr/codestyle.yml:17-30 Suggested fix:
Related: PR #11796; checker introduced/last modified by #11527; style rules at https://github.com/openucx/ucx/wiki/Guidance-for-contributors#general-guidelines
|
|
/build |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage Root cause: Implicated commit: File: Suggested fix: Rewrite the offending commit's title so it satisfies the convention, then force-push the branch: Since the commit is described as a throwaway debug change, the cleaner option is to drop it entirely or squash it into the adjacent Related: PR #11796 (this PR); prior commit-title-check failures: #11218, #11480, #8246. Range-limiting logic last touched by
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage Root cause: Evidence from the stage log (node 62 / 37): The modulefile itself exists and is readable ( This is environment-specific, not the PR: the parallel Implicated commit: unknown — no UCX commit implicated. The File: Suggested fix:
function az_module_load() {
local module=$1 retries=5
until [ -e "/hpc/local/etc/modulefiles/${module}" ] || \
module avail -t 2>/dev/null | grep -q "^${module}\$"; do
...retry/automount refresh...
done
module load "$module" || { azure_log_warning "Module $module cannot be loaded"; return 1; }
}
Related: #11796 (PR under test; unrelated to this failure) |
|
🤖 CI Triage Agent — TL;DR: Both Coverity stages failed because Full analysisSummary: Root cause: Environment-infrastructure failure, not a code bug in UCX:
Implicated commit: [REDACTED:Hex High Entropy String] — Noam Tsemah, "debug CI failure" (adds the File: Suggested fix:
Related: PR #11796 (
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 160 "Checkout & test" (aarch64/build_helper_dl) hung in the UCP client-server example loop and was manually aborted; the abort/ Root cause: Log timestamps show continuous progress through all The client never printed even This is the first Implicated commit: none — PR #11796 / [REDACTED:Hex High Entropy String] is a CI-URL-only change and is not implicated. No recent commits to File: Suggested fix:
Related: PR #11796 (the PR under test — CI URL change only). No existing issue found for a
|
|
🤖 CI Triage Agent — TL;DR: All four gtest shards failed on four different nodes (funk06/11/12/27) because UCP connection establishment over the nodes' own IB/IPoIB addresses (1.1.90.x / 1.1.91.x) returns "Destination is unreachable" — a broken fabric/IPoIB state on the allocated GPU test nodes, not a code change. Fix the node fabric (or drain those nodes) and add a fail-fast IB/IPoIB health check plus a Full analysisSummary: Root cause: Environmental — RDMA connectivity on the allocated nodes is broken while the IB ports themselves enumerate as active. Evidence:
The PR under test cannot explain this: PR #11796 is "CI: Replace Blue Ocean with pipeline-overview URL" and the only branch commit, 08e1ede ("debug CI failure", Noam Tsemah), touches Implicated commit: none — not a source regression. Branch commit 08e1ede (Noam Tsemah, "debug CI failure", File: Suggested fix:
Related: PR #11796 (the triggering PR — CI URL change only, unrelated to the failure); branch commit 08e1ede indicates the CI owner was already debugging this environment. |
|
🤖 Starting review — findings will be posted here when done. |
|
/build |
|
Test/validation coverage note: Per REVIEW.md, build/CI changes should show the intended CI behavior changed. The URL change is low-risk and self-evident, but there's no dry-run/CI output demonstrating that the pipeline-overview link resolves correctly. Worth confirming the link works before merge, though it does not block. |
|
🤖 Starting review — findings will be posted here when done. |
|
/build |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — TL;DR: The build failed in the Slurm Full analysisSummary: Stage Root cause: The pipeline requested an exclusive 1-node allocation on the GB300 CI partition: The ~60-minute log gap (12:36:11 → 13:36:20) is exactly the Secondary infra noise, non-fatal: the build_helper_dl agent pod was initially rejected three times with Implicated commit: none — commit f0fcdd6 / PR #11796 ("CI: Replace Blue Ocean with pipeline-overview URL") only changes a CI URL and cannot affect Slurm allocation. Pipeline library used was swx-jenkins-lib@8a2b2c6d ("slurm: verify scctl installer checksum before executing it"), also unrelated to allocation logic. File: not in this repo — the failing step is the Suggested fix:
Related: #11796 (the PR under test; unrelated to the failure) |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Build #8 failed only in parallel branch Root cause: Infrastructure, not code. The log shows a completely healthy build: Corroborating evidence that this is environmental:
Implicated commit: none — the PR commit is not at fault (same commit passed on 3 of 4 parallel workers). No UCX source file is implicated. File: unknown (no source file implicated; failure originates in the Jenkins Kubernetes agent transport, surfaced at stage node 259 of Suggested fix:
Related: none found — a repository search for |
|
🤖 CI Triage Agent — TL;DR: This build did not fail on a code or test defect — it was manually aborted mid-run ("Aborted by Noam Tsemah") while the gtest suite was passing, and the resulting SIGTERM cascade killed the Full analysisSummary: Root cause: Manual cancellation, not a test or build error.
Secondary, real robustness bug exposed: because Implicated commit: none for the failure itself. Build tested File: Suggested fix:
Related: #11796 (the PR being built; CI-URL-only change, unrelated to the abort) |
|
🤖 CI Triage Agent — TL;DR: The build wasn't a code failure — the Full analysisSummary: Stage Root cause: The log shows continuous, fast progress through all
That is a ~2 hour 9 minute gap with zero output — the very first The failure is unrelated to the commit under test: PR #11796 is "CI: Replace Blue Ocean with pipeline-overview URL", a Jenkins-URL-only change with no effect on UCX runtime or on the stream code path. Contributing CI weakness: Implicated commit: unknown — not caused by f0fcdd6 (PR #11796, CI URL change only). No recent commits to File: Suggested fix:
Related: #11796 (the PR under test — CI URL change, not the cause) |
|
🤖 CI Triage Agent — TL;DR: All four GPU test workers died because Full analysisSummary: Root cause: A hang in the UCP RMA GET completion path, not a slow test. Timestamp gaps prove it:
Both failures are the last, largest size step of PR #11796 (branch Implicated commit: File: Suggested fix:
Related: PR #11808 (#11808) — suspected regression source; PR #11796 (#11796) — the CI-only PR that merely surfaced it. Adjacent FT/RMA lane work worth reviewing: #11563 (
|
|
🤖 CI Triage Agent — TL;DR: The dispatcher didn't fail on its own — Full analysisSummary: Dispatcher build #5 reported all 8 downstream stages as ABORTED at an identical ~200 min; 6 succeeded, Root cause: Two independent, unrelated issues, both symptomatic of wall-clock kills rather than real test assertions:
Implicated commit: None in UCX code. Trigger commit [REDACTED:Hex High Entropy String] / PR #11796 is "CI: Replace Blue Ocean with pipeline-overview URL", a CI URL-string change that cannot affect UCP stream data transfer. File: Suggested fix:
Related: PR #11796 — CI: Replace Blue Ocean with pipeline-overview URL (trigger only, not the cause). No existing issue found for the
|
f0fcdd6 to
2338720
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Test/validation coverage note: Consistent with the prior review, this is a CI-config change with no dry-run/CI output demonstrating that the |
Replace GitHub commit status links to point to Jenkins' pipeline-overview page instead of Blue Ocean UI. Signed-off-by: Noam Tsemah <ntsemah@nvidia.com>
Signed-off-by: Noam Tsemah <ntsemah@nvidia.com>
2338720 to
feec90b
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — TL;DR: The GPU gtest job hung in Full analysisSummary: Stage Root cause: Evidence from the stage log (build #12, node 298):
Implicated commit: 84548e4 — "UCP/RMA: Support failover for offloaded GET bcopy protocol (#11840)", Evgeny Leksikov (not the PR commit feec90b, which is CI-only) File: src/ucp/rma/get_offload.c:48-56 ( Suggested fix:
Related: #11840 (implicated change), #11796 (PR under test, CI-only), earlier failover work #11266 and #11162 |
|
🤖 CI Triage Agent — TL;DR: The build was not slow — it hung: Full analysisSummary: Stage Root cause: From the stage log, all
That is a 2 h 09 min gap with zero output — by far the largest gap in the log and essentially the whole runtime. The client never even printed A contributing CI defect: The PR under test (#11796 "CI: Replace Blue Ocean with pipeline-overview URL", branch HPCINFRA-4739, commit feec90b) only changes a Jenkins URL and cannot affect the UCP stream path — so the hang is a pre-existing/environmental flake, not introduced by this PR. Implicated commit: unknown — not caused by [REDACTED:Hex High Entropy String] (CI-URL-only change). File: Suggested fix:
Related: PR #11796 (build under test, CI-only change); no existing issue found for a
|
|
The Jenkins builds were triggered by ucx-dispatcher-oss-test, a test dispatcher job running with this PR's changes. |
What?
Replace GitHub commit status links to point to Jenkins' pipeline-overview page instead of Blue Ocean UI.
Why?
Justification for the PR. If there is an existing issue/bug, please reference it. For
bug fixes, the 'Why?' and 'What?' can be merged into a single item.
How?
It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.