ci: fix build-container push step auth and silent curl failure

[NirWolfer]

Summary

• The Artifactory property-setting curl call used Authorization: Bearer which doesn't match the usernamePassword (svc-nixl-new-artifactory-token) credential type — docker push was succeeding but the subsequent REST API call to set image properties was returning 401 on every run
• curl exits 0 on HTTP errors by default, and the Push step had no set -e, so the 401 was printed and silently swallowed — the step passed green while properties were never set

Changes

• Auth fix: switched curl from Authorization: Bearer ${ARTIFACTORY_PASSWORD} to -u "${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}" (basic auth, matching the credential type)
• Failure propagation: added --fail to curl so HTTP 4xx/5xx exits non-zero; added set -eo pipefail to the Push step so that exit propagates and fails the Jenkins step

Summary by CodeRabbit

Summary by CodeRabbit

• Bug Fixes

  • Improved container publishing reliability so authentication or API errors abort the build immediately.
  • Tightened failure handling for pushing image build metadata to ensure HTTP errors stop the pipeline.

• Documentation

  • Clarified that the CI workflow pushes container images and then sets build metadata properties via the Artifactory REST API, with failures stopping the job.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c139526d-af31-4ba0-b65a-75ed38214119

📥 Commits

Reviewing files that changed from the base of the PR and between 5a1f747 and 7606cfc.

📒 Files selected for processing (2)

• .ci/docs/ci-overview.md
• .ci/jenkins/lib/build-container-matrix.yaml

---

📝 Walkthrough

Walkthrough

The Jenkins container push step now propagates shell, pipeline, and Artifactory HTTP failures. CI documentation describes the metadata update and fail-fast behavior.

Changes

Container push error handling

Layer / File(s)	Summary
Push failure handling and documentation .ci/jenkins/lib/build-container-matrix.yaml, .ci/docs/ci-overview.md	The push step enables set -eo pipefail, the Artifactory metadata request uses curl --fail with its existing bearer authentication, and the documentation reflects these behaviors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: dpressle

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR's main fix: Artifactory auth and curl failure handling in the build-container push step.
Description check	✅ Passed	It includes the needed what/why/how details, but uses Summary/Changes instead of the template's What/Why/How headings.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/lib/build-container-matrix.yaml:
- Around line 181-182: Update the curl invocation in the image publication step
to avoid embedding ARTIFACTORY_PASSWORD in command-line arguments. Create a
temporary credentials file with 0600 permissions, use it via curl’s --netrc-file
or --config option, and register an EXIT trap to remove the file, while
preserving the existing authenticated PUT request.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ddd118f0-c383-4757-b015-6b1ab3b7ea2f

📥 Commits

Reviewing files that changed from the base of the PR and between cdb4252 and 72bc766.

📒 Files selected for processing (2)

• .ci/docs/ci-overview.md
• .ci/jenkins/lib/build-container-matrix.yaml

[NirWolfer]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit ecbb2c1b

TL;DR: The "Run vLLM sanity" stage (#518) failed because the model prefetch huggingface_hub.snapshot_download('Qwen/Qwen3-8B') got an HTTP 429 "Too Many Requests" rate limit from HuggingFace on an anonymous shared CI IP — an external/infra issue, not a PR code defect. Fix by passing an authenticated HF_TOKEN (and/or caching weights) so the prefetch isn't anonymously rate-limited.

Full analysis

Summary: nixl-ci-build-wheel #1187 failed in the "Run vLLM sanity" pipeline stage (node 518); the aarch64 wheel build itself completed successfully (the "FAILURE" marks on stages 332/333-parallel are the downstream propagation of this sanity failure, not build errors).

Root cause: In .gitlab/test_vllm_sglang_sanity.sh line 99, the script prefetches the sanity model via snapshot_download('Qwen/Qwen3-8B') with no HF token. HuggingFace returned 429 Too Many Requests: you have reached your 'api' rate limit ... We had to rate limit your IP (216.228.125.128) ... make sure you pass a HF_TOKEN. The unauthenticated request from a shared CI egress IP hit the anonymous rate limit, so the model download raised LocalEntryNotFoundError and the script exited with code 1 (set -euo pipefail). The failure was immediate (job started ~12:30:50, error at ~12:32:17, right after "prefetching Qwen/Qwen3-8B") — not a hang or timeout.

Implicated commit: unknown (not a code regression; the failure is external HF rate-limiting). The prefetch logic lives in .gitlab/test_vllm_sglang_sanity.sh.

File: .gitlab/test_vllm_sglang_sanity.sh:99

Suggested fix: Make the model prefetch resilient to HF anonymous rate limits:

1. Authenticate the download — set HF_TOKEN in the CI environment (a secret) so snapshot_download is not subject to the anonymous per-IP limit. Do NOT hardcode the token in the script.
2. Add retry/backoff — wrap the snapshot_download call in a retry loop honoring the Retry after N seconds hint (429s here said retry after ~236s), or pass max_retries/use tenacity, so a transient rate limit doesn't fail the whole job.
3. Prefer a persistent/pre-populated HF cache (e.g. a shared HF_HOME volume on the SLURM node or bake weights into the base image) so repeated CI runs don't re-hit the Hub. This is an infra retry — safe to re-run the build once the rate-limit window clears.

Related: none found.

[NirWolfer]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 326d53fe

TL;DR: The aarch64 container build (stage 233, container-build/nixl/aarch64) failed at Dockerfile STEP 23 because apt-get upgrade couldn't fetch glibc .debs (libc6, libc6-dev, libc-bin, libc-dev-bin 2.39-0ubuntu8.8) — the Ubuntu ports mirror returned 404 Not Found. This is a transient upstream mirror/stale-index issue, not a code bug; re-run the build (and ideally add apt-get update retry hardening).

Full analysis

Summary: aarch64 NIXL container image build failed at Dockerfile STEP 23/62 (DOCA host install + apt-get upgrade) with subprocess exited with status 100.

Root cause: apt-get upgrade -y tried to pull glibc packages at version 2.39-0ubuntu8.8 from http://ports.ubuntu.com/ubuntu-ports .../noble-security/main, but those files returned 404 Not Found (E: Failed to fetch .../libc6-dev_2.39-0ubuntu8.8_arm64.deb 404 Not Found, same for libc-dev-bin, libc6, libc-bin). The apt index referenced a security point-release that had already been superseded/removed on the mirror — a classic race between a stale package list and a mirror rotation. The x86_64 build passed, confirming this is arch/mirror-specific transience, not a repo change. Notably the apt-get update in this step ran against a Hit: (cached) index rather than a fresh fetch, so the stale version list was reused.

Implicated commit: none — not caused by a code commit (326d53f / PR #1997 is unrelated; recent Dockerfile history shows no change to STEP 23).

File: contrib/Dockerfile — STEP 23 (RUN ... dpkg -i doca-host.deb && apt-get update && apt-get upgrade -y ...)

Suggested fix:

1. Retry the build — the mirror will normally re-sync and the 404s disappear.
2. To harden against recurrence, in STEP 23 make the update/upgrade resilient, e.g.:
   • Ensure a real refresh before upgrade: apt-get update --error-on=any (or drop cached lists with rm -rf /var/lib/apt/lists/* && apt-get update) so apt doesn't reuse a stale noble-security index.
   • Add retry/tolerance: wrap apt-get upgrade in a small retry loop, or consider whether a full apt-get upgrade -y is even needed here (scoping the upgrade to only DOCA-required packages would avoid pulling unrelated glibc security updates that are prone to this mirror race).

Related: none found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 326d53fe

TL;DR: The Run vLLM sanity stage failed because the model-weights prefetch step got an HTTP 429 "Too Many Requests" rate-limit from HuggingFace Hub when downloading Qwen/Qwen3-8B anonymously — an infrastructure/rate-limit issue, not a code bug; fix by authenticating the download with an HF_TOKEN (and/or caching weights on the node).

Full analysis

Summary: The vLLM sanity test (stage 535, .gitlab/test_vllm_sglang_sanity.sh vllm) aborted at the model prefetch step with exit code 1.

Root cause: At line 99 of test_vllm_sglang_sanity.sh, snapshot_download('Qwen/Qwen3-8B') is called with no HF token. HuggingFace returned 429 Too Many Requests: you have reached your 'api' rate limit … We had to rate limit your IP (216.228.125.128) … make sure you pass a HF_TOKEN if you're using the API. huggingface_hub raised LocalEntryNotFoundError and the set -euo pipefail script exited non-zero. This is an external rate-limit on the shared CI egress IP, not a defect in the PR. (The build_helper_vllm stage 332 failure was a fail-fast propagation of the same parallel branch, not an independent build error — the wheel built and audited successfully.)

Implicated commit: unknown — not a code regression; caused by external HF Hub rate limiting of the CI node's shared IP.

File: .gitlab/test_vllm_sglang_sanity.sh:99

Suggested fix: Authenticate the prefetch so it isn't subject to the anonymous per-IP rate limit: inject a HF_TOKEN secret in the CI environment (huggingface_hub picks it up automatically), and make the download resilient — e.g. add retry/backoff on 429 and prefer a pre-populated/shared HF_HOME cache on the SLURM node so repeated CI runs don't re-hit the API. Rerunning the job after the 300s window resets will also pass, but adding the token is the durable fix. No secret token values were present in the logs.

Related: none found.

[NirWolfer]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 34da9ff2

TL;DR: The Test CPP stage failed because the gtest ucx/TestErrorHandling.XferThenFail/0 crashed with SIGSEGV during the UCX error-injection/teardown path; it's a crash in the UCX backend error-handling flow, not a timeout or hang. Fix by investigating the null-deref/use-after-free in the UCX endpoint failure-handling path exercised by that test.

Full analysis

Summary: Jenkins nixl-ci-non-gpu #2475 failed in the x86_64 cuda12.9 Test CPP stage (node 417) when the gtest ucx/TestErrorHandling.XferThenFail/0 was killed by SIGSEGV (exit code 245).

Root cause: A segmentation fault in the UCX backend's transfer-then-fail error-handling code path. The test first passed at [85/154] (581 ms) and gtest-parallel then re-ran it in isolation, where it was killed by signal SIGSEGV (581 ms). The log shows continuous test progress with no multi-minute gaps, so this is a genuine crash (likely a use-after-free / null-pointer deref when tearing down or draining a UCX request after an injected endpoint failure), not a wall-clock hang. All other 153 tests passed. (Stage 300's setup "FAILURE" was auto-retried successfully as stage 357 and is not the cause.)

Implicated commit: Not definitively from logs. Most likely candidates touching this path recently: 0225cd5a0923 (Eylon Eliyahu Krause, "ucx: fix infinite loop dropping pending requests on dedicated-thread exit", #1880) and 22cdab3de2e9 (Raul Akhmetshin, "Added option to forcibly close UCP endpoints", #1846) — both alter UCX request/endpoint teardown, which is exactly what XferThenFail exercises.

File: UCX backend error-handling path exercised by ucx/TestErrorHandling.XferThenFail (in src/plugins/ucx/ — request/endpoint teardown after failure). Exact path:line unavailable; the crash produced no stack trace in the captured log.

Suggested fix: Reproduce locally with gtest-parallel --workers=1 --repeat=N ./bin/gtest --gtest_filter='ucx/TestErrorHandling.XferThenFail/0' under a debugger or ASan (the repo already has ASan CI builds from #1709) to capture the crashing frame. Audit the post-failure request cleanup in the UCX engine for use-after-free/double-free of the transfer handle or endpoint after XferThenFail injects the error, and add a null/validity guard before dereferencing the request/endpoint during teardown.

Related: PR #1880 (UCX pending-request drop on thread exit), PR #1846 (forcibly close UCP endpoints), PR #1743 (proposal to run gtest in a single process — relevant to the parallel/retry behavior seen here). No existing issue matches this specific SIGSEGV.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 34da9ff2

TL;DR: The "Run DL CPP tests" stage was killed (exit 143 / SIGTERM) after ucx_backend_test hung in the inter-agent DRAM→VRAM READ (non-inline rndv GET) test — each READ's completion/data-verification stalled for 10–14 minutes instead of milliseconds — so the fix is to investigate the UCX GET/read completion path, not to raise the timeout.

Full analysis

Summary: Jenkins job nixl-ci-dl-gpu #1658 stage "Run DL CPP tests" (node 336) was ABORTED after ~50 min because ./bin/ucx_backend_test hung.

Root cause: A hang, not a slow job. Timestamps show the test ran normally with sub-second gaps up through the Inter-agent ... P-Thr=OFF, Handler-reuse=ON, (DRAM -> VRAM) READ (10 iterations) section. The last healthy line is at 15:57:15.363. After that, consecutive READ iterations stall enormously:

• 15:57:15 → 16:07:07 (~10 min gap)
• 16:07:07 → 16:21:29 (~14 min gap)
• 16:21:29 → 16:35:36 (~14 min gap)
• then SIGTERM at 16:42:57 ("Sending interrupt signal", "Terminated", script returned exit code 143).

Each non-inline (rendezvous) GET/READ in the DRAM→VRAM handler-reuse path is blocking for many minutes on completion, i.e. the UCX GET request never (or barely) completes and the test's poll/verify loop wedges. This is a functional hang in the UCX backend's rendezvous GET path exercised with handler reuse + cross-memory-type (host DRAM → CUDA VRAM) transfers.

Implicated commit: Not definitively proven from logs alone, but the prime suspect is #1917 [REDACTED:Hex High Entropy String] — "PLUGINS/UCX: Do not use strict order key for PUT/GET" (Thomas Vegas, 2026-07-21), which directly changed the PUT/GET (i.e. WRITE/READ) protocol path that is now hanging, and landed just before this build. Related recent UCX rndv change: #1854 [REDACTED:Hex High Entropy String] "Allow rndv-based PUT/GET protocol without error handling". This build is running under PR #1997 — confirm whether #1997 also touches the UCX transfer path.

File: test/unit/plugins/ucx/ucx_backend_test.cpp (the hanging READ loop) and the GET/read completion path in src/plugins/ucx/ucx_backend.cpp (rndv GET progress/completion). Hang site is the "READ test" iteration in the Handler-reuse=ON, (DRAM -> VRAM) block.

Suggested fix: Do not raise the stage time limit. Investigate the UCX rendezvous GET completion path for DRAM→VRAM transfers with handler reuse:

1. Bisect against PLUGINS/UCX: Do not use strict order key for PUT/GET #1917 (strict-order-key removal for PUT/GET) and Allow rndv-based PUT/GET protocol without error handling #1854 (rndv PUT/GET without error handling) — revert/guard PLUGINS/UCX: Do not use strict order key for PUT/GET #1917 and re-run the CPP tests to confirm the READ path no longer stalls.
2. Add a per-transfer watchdog/timeout in ucx_backend_test so a stuck GET fails fast with diagnostics instead of hanging the whole stage for ~45 min.
3. Check whether the ordering-key change interacts badly with rndv GET completion when source is host memory and destination is CUDA memory (the only variant that hung).

Related: PR #1917 (#1917); PR #1854; the failing run is PR #1997 / build #1658.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id dcf388e1-9d55-4ae5-bd93-8d242772958c in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 34da9ff2

TL;DR: The two GPU "Allocate Environment" stages were killed (exit 143/SIGTERM) after ~58 minutes because the SLURM salloc on partition gb200nvl72_ci never got a node — the retry logged job queued and waiting for resources. This is a cluster GPU-capacity/scheduling issue, not a defect in PR #1997's code.

Full analysis

Summary: Jenkins aborted the build_helper_vllm and build_helper_sglang GPU pipelines because their salloc GPU allocation requests hung waiting for SLURM resources and were killed at the build wall-clock limit.

Root cause: In both stages the salloc -N 1 -p gb200nvl72_ci ... --immediate=3600 command ran at 15:47:34 and produced zero output for ~58 minutes (single largest gap = essentially the entire stage runtime) until Jenkins sent SIGTERM at 16:45:59 (exit code 143). This is not a code hang — it is SLURM queuing: the retry immediately after the kill printed salloc: Pending job allocation 1671450/1671451 and salloc: job queued and waiting for resources, confirming the gb200nvl72_ci partition had no free GB200 nodes (2× GPU, 24 CPU, 128G requested). The upstream build/wheel stages all succeeded; only the GPU-node acquisition failed. The --immediate=3600 (1h) SLURM timeout also outlives the ~58min Jenkins step wall clock, so Jenkins kills the job before SLURM's own immediate timeout can fail it cleanly.

Implicated commit: unknown — not caused by commit 34da9ff; this is an infrastructure/capacity failure in the SLURM gb200nvl72_ci partition.

File: N/A (Jenkins/SLURM allocation via slurm.allocation in swx-jenkins-lib, not repo source)

Suggested fix: Treat as infra: check gb200nvl72_ci partition capacity/queue on dlcluster.nvidia.com and retry the build once nodes are free. Also fix the timeout misconfiguration so SLURM fails fast instead of hanging: set --immediate (immediateTimeout) shorter than the Jenkins stage wall-clock (currently 3600s immediate vs ~58min Jenkins kill), so a "no resources" condition returns a clean salloc error rather than a SIGTERM abort. Consider a bounded auto-retry/backoff on the allocation step. Do not raise the wall-clock limit — the process was waiting in the SLURM queue, not making progress.

Related: none found (searched issues/PRs for the allocation timeout signature; no matching report).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 34da9ff2

TL;DR: The build itself passed; the pipeline failed in "Allocate DL Environment" because salloc on the gb200nvl72_ci Slurm partition waited its full --immediate=3600 (1h) window and never got a node ("queued and waiting for resources" → "Unable to allocate resources: Connection timed out"). This is a cluster capacity/infra issue, not a PR code defect — re-run once GB200 nodes are free.

Full analysis

Summary: Jenkins stages "Allocate DL Environment" (node IDs 200 and 183) failed after the Slurm salloc request for a gb200nvl72_ci node timed out.

Root cause: Resource starvation on the GPU cluster. At 17:01:00 the job issued salloc -N 1 -p gb200nvl72_ci --immediate=3600 ... --account=oberon-gb-ci; Slurm queued it (Pending job allocation 1671826, queued and waiting for resources) and exactly one hour later (18:01:08) returned salloc: error: Unable to allocate resources: Connection timed out. No GB200 node became available within the 1-hour immediate window. The ~60-minute gap between the two log lines is Slurm blocking on the queue, not a hung application — this is legitimately waiting for scheduler resources that never freed up.

Implicated commit: none — not caused by commit 34da9ff / PR #1997. The Docker image compiled, linked, installed, and pushed successfully.

File: N/A (CI infrastructure / Slurm allocation step, slurm.allocation in the Jenkins pipeline library swx-jenkins-lib)

Suggested fix: Re-trigger the build when the gb200nvl72_ci partition has free nodes; this is transient cluster capacity, not a code problem. If GB200 CI allocation timeouts are recurring, consider (a) raising immediateTimeout beyond 3600s, or (b) adding automatic requeue/retry of the "Allocate DL Environment" stage on Unable to allocate resources, and (c) checking partition health/queue backlog with the cluster admins. Note: this is one of the rare cases where the wait was genuine queueing, not a hang, so a longer/retried allocation window is appropriate.

Related: none found.