CI: Build containers nightly on the Pytorch release image

[Alexey-Rivkin]

What

nixl-ci-build-container nightly cron in proj-jjb.yaml:

• All 4 entries: H 3 / H 4 -> H 22 (21:00 CET, as requested)
• PyTorch entries: internal DLFW gitlab-master.nvidia.com:5005/dl/dgx/pytorch:main-py3-devel -> official release nvcr.io/nvidia/pytorch:26.06-py3
• ci-overview.md trigger line updated to match

Why

The NIXL team asked to move the nightly to 21:00 CET and base the pytorch builds on the latest official release image. The schedule was changed via the Jenkins UI at the time; this lands both changes in the JJB definition so the next deploy does not revert them.

nvcr.io/nvidia/pytorch:26.06-py3 is the same image the PR container gate already builds against, so the nightly and the gate now use one base.

Nightly builds over the cuda-dl-base image are untouched - still built every night.

Notes

• Published tag names embed BASE_IMAGE_TAG, so the pytorch nightly tags change from 26.06-py3-devel-* to 26.06-py3-* (incl. the -latest aliases) once deployed.
• Takes effect on the next JJB deploy.

Summary by CodeRabbit

• CI/Build Updates
  • Nightly container builds now run around 22:00 instead of 03:00–04:00.
  • Builds use the PyTorch release image nvcr.io/nvidia/pytorch:26.06-py3 for both NIXL targets.
  • Updated documentation reflects the new image and schedule.

[github-actions]

👋 Hi Alexey-Rivkin! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[ntsemah]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 3d55df12

TL;DR: The build didn't fail from a code defect — all parallel container-build stages were manually aborted by a user (ntsemah@nvidia.com) while still pulling the base image, producing exit code 143. No code fix is needed; simply re-run the build.

Full analysis

Summary: All parallel "Build image" stages (nixl/nixlbench × x86_64/aarch64) terminated with exit code 143 via FlowInterruptedException: Aborted by ntsemah@nvidia.com.

Root cause: A human manually cancelled the Jenkins build. Every failing stage was interrupted during the very first Dockerfile step — FROM nvcr.io/nvidia/pytorch:26.06-py3 / "Trying to pull...". The logs show continuous, active blob-copy progress right up to the "Sending interrupt signal to process" line (e.g. blobs copying at 09:04:33–09:04:36, interrupt at 09:04:36.781 for x86_64; similar for the others). There is no silent gap, so this is not a hang or a wall-clock timeout — it is an operator-initiated abort. The build never reached any NIXL/nixlbench source-compilation step, so nothing in PR #2061 was exercised or implicated.

Implicated commit: none — the failure is unrelated to commit [REDACTED:Hex High Entropy String]; the build was cancelled before any repo code was built.

File: none (interruption occurred during base-image pull, before any project file was compiled).

Suggested fix: No code change required. Simply re-trigger nixl-ci-build-container-pr for PR #2061. If the abort was intentional (e.g. superseded by a newer push), it can be ignored. If cancellations are recurring, confirm with ntsemah@nvidia.com why the run was stopped (e.g. queue/resource management or a duplicate run).

Related: none

(Note: the repeated "detected dubious ownership in repository" git warnings are benign noise from a UID mismatch on the workspace and did not cause the failure, though adding git config --global --add safe.directory to the agent setup would clean up the log.)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 6032f612-ca92-4485-b9a8-eb6a4dbd351b in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 3d55df12

TL;DR: The ASAN/UBSAN gtest suite failed on a single test, ucx_tracing_no_pt/TestTransferTracing.NvtxNotifications/0, which received 0 of 4 expected notifications after spinning for ~101s — the no-progress-thread notification path never drained the UCX active-message sends, which were then cancelled at teardown (UCX AM send failed with status -16 (Request canceled)).

Full analysis

Summary: nixl:sanitizer / gtest failed (exit 42) with 1 failed test out of 206; all smoke tests passed — the stage-level "Sanitizer test FAILURES" message is a downstream consequence of that one gtest failure.

Root cause: In TestTransferTracing.NvtxNotifications run under the ucx_tracing_no_pt variant (progress thread OFF), notification delivery depends on the test manually driving UCX progress via getNotifs(). The generated notifications (genNotif) never completed — notif_list.size() was 0 vs. the expected 4 at test/gtest/test_transfer.cpp:334 (EXPECT_EQ(notif_list.size(), expected_count)), and the test consumed the full retry budget (~101 s) before the AM sends were cancelled during endpoint invalidation. The pt (progress-thread-on) variant of the same test passed in 137 ms, and 185/206 tests passed — pointing to a flaky/timing-sensitive progress-draining bug in the no-progress-thread NVTX notification path rather than a broad regression. This is the NVTX-tracing test path added in PR #1852/#1845 (e-eygin), with additional UCX cases added just before this build in PR #2059.

Implicated commit: No single definitive commit; the failing test path was introduced by 471a64e / 765f2f2 (e-eygin, NVTX tracing tests). Most recent touch to the file: 563aaa5 "GTEST/UCX: Added more test cases for UCX (#2059)" (Ilia Yastrebov, 2026-08-13).

File: test/gtest/test_transfer.cpp:334 (assertion in verifyNotifs); notification-drain logic at test/gtest/test_transfer.cpp:360-373 (doNotificationTest).

Suggested fix: In the no_pt path of doNotificationTest/verifyNotifs, ensure UCX progress is driven on both the sender (from) and receiver (to) agents while waiting — the current verifyNotifs retry loop only calls to.getNotifs(), so pending AM sends on from may never complete without a progress thread. Add a from.getNotifs()/progress call inside the verifyNotifs retry loop for the no-progress-thread case (mirroring the pattern already used in doTransfer), and confirm the retry budget is adequate under sanitizers. Re-running the build will confirm whether it's an intermittent timing failure.

Related: none (no existing tracking issue; PR #2059 recently modified the file).

[Alexey-Rivkin]

/build

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 452d451d-38f5-4164-ba48-7bfa656e6f18

📥 Commits

Reviewing files that changed from the base of the PR and between 8585eee and c9fc032.

📒 Files selected for processing (2)

• .ci/docs/ci-overview.md
• .ci/jenkins/pipeline/proj-jjb.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The nightly container-build schedules now run at approximately 22:00. PyTorch release builds use nvcr.io/nvidia/pytorch:26.06-py3. The CI documentation reflects these changes.

Changes

Nightly container builds

Layer / File(s)	Summary
Align nightly schedules and documentation .ci/jenkins/pipeline/proj-jjb.yaml, .ci/docs/ci-overview.md	Nightly schedules use 22:00 runs with Ubuntu 24.04 defaults and the PyTorch 26.06 release image. Documentation reflects the updated image and schedule.

Estimated code review effort: 2 (Simple) | ~5 minutes

Suggested reviewers: nirwolfer

Merge Risk: ⚪ Minimal · up to c9fc0

Nightly container builds move to the PyTorch 26.06 release image and approximately 22:00 schedule, with matching documentation. No current merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the primary change: nightly container builds now use the PyTorch release image.
Description check	✅ Passed	The description includes complete What and Why sections, plus relevant implementation notes. The optional How section is not required for this change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit c9fc0320

TL;DR: The ASAN/UBSAN sanitizer variant failed because the gtest MDManagerEtcdFixture.InvalidateLocalRemovesRemote case never saw the remote metadata disappear — the ETCD backend arms its agent watcher without a start revision and only after loadRemoteMD has already published the metadata, so a DELETE that lands before the watch stream is established is lost forever. Fix: create the agent watcher with the revision returned by the preceding get() (and before publishing the fetched MD), so the invalidation event cannot be missed.

Full analysis

Summary: Stage "Test Sanitizer" (asan_ubsan variant, node id 177) failed at 1/1 nixl:sanitizer / gtest FAIL 43.74s with one failing test: MDManagerEtcdFixture.InvalidateLocalRemovesRemote (../test/gtest/md_manager.cpp:298, checkRemoteMD returned NIXL_SUCCESS (0) instead of NIXL_ERR_NOT_FOUND (-4)). The tsan variant of the same suite passed.

Root cause: A lost-watch-event race in the ETCD metadata backend, not a test-timeout-only issue:

• nixlEtcdMetadataBackend::fetchRemote calls ctx_.loadRemoteMD(blob, …) first (line 401) and only then client_->setupAgentWatcher(remote_name) (line 413). The test's waitForRemoteMD(..., NIXL_SUCCESS) therefore returns as soon as the MD is loaded — before the watcher exists — and immediately proceeds to src.agent->invalidateLocalMD(nullptr).
• setupAgentWatcher builds etcd::Watcher(*etcd_, agent_prefix, process_response) with no start index, so it only observes revisions created after the gRPC watch stream is up. Contrast waitForMetadataFromEtcd, which correctly passes response.index() as watch_index (line 203/229).
• invalidateLocal → removeMetadataFromEtcd does etcd_->rmdir(agent_prefix, true). If that range delete commits inside the window between the fetch and the watch being established (ASAN slows the process by several-fold, widening it), no DELETE event is ever delivered, invalidatedAgents_ stays empty, processInvalidatedAgents never calls ctx.invalidateRemoteMD, and checkRemoteMD keeps returning NIXL_SUCCESS for the full 3 s poll — matching the test's 3593 ms duration and the absence of any watcher/invalidate error lines in the captured stderr.
• Secondary robustness bug in the same callback: if (response.events().size() != 1) { … return; } (line 278) discards the whole notification. A prefix rmdir deletes both the anchor key and the metadata key, so a batched multi-event response is legitimate and would also be dropped.

Implicated commit: 2ff0105f — "Metadata manager pr5 (#1930)", aschwartz12 (the only commit touching both src/core/nixl_etcd_metadata_backend.cpp and test/gtest/md_manager.cpp). Note the triggering PR #2061 only changes proj-jjb.yaml/ci-overview.md, so it is not the cause.

File: src/core/nixl_etcd_metadata_backend.cpp:262-299 (setupAgentWatcher), used from src/core/nixl_etcd_metadata_backend.cpp:401-413 (fetchRemote); test assertion at test/gtest/md_manager.cpp:298.

Suggested fix:

1. Arm the watcher at a known revision. Have fetchOrWaitForMetadataFromEtcd return the revision it read (the response.index() already available in fetchMetadataFromEtcd/waitForMetadataFromEtcd) and pass it through: std::make_unique<etcd::Watcher>(*etcd_, agent_prefix, start_index, process_response, /*recursive=*/true). That makes the DELETE observable even if it commits before the stream is up.
2. Arm the watcher before ctx_.loadRemoteMD(...) in fetchRemote, so the metadata is never visible to callers while unwatched; on load failure, drop the watcher again.
3. Make the callback tolerant of batched events: iterate response.events() and enqueue the agent on any DELETE_ event instead of bailing out when size() != 1.
4. Belt-and-braces: after arming the watcher, re-get the agent prefix; if the key is already gone, enqueue the agent for invalidation immediately.

Related: PR #1930 (2ff0105f, "Metadata manager pr5") introduced this code path; triggering PR is #2061 (CI-config only, unrelated to the failure).

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit c9fc0320

TL;DR: The newly added test/python/test_tcpstore_metadata.py::test_tcpstore_metadata_exchange failed on the aarch64 non‑GPU variant because its PyTorch TCPStore master could not be reached on the CI-preassigned port 10507 within its hard-coded 5 s timeout; make the test use a kernel-assigned port (port=0, it already reads tcp_store.port) and a larger/retried timeout.

Full analysis

Summary: Stage "Test Python" (node 574, aarch64/ubuntu2204 variant) failed with torch.distributed.DistNetworkError: The client socket has timed out after 5000ms while trying to connect to (127.0.0.1, 10507); 1 failed, 24 passed, 3 skipped. The other five parallel Test Python stages passed.

Root cause: Test-harness fragility, not a nixl product defect. .gitlab/test_python.sh:70 preassigns NIXL_TCPSTORE_PORT via get_next_tcp_port (.ci/scripts/common.sh:38-61), whose availability check is ss -tuln | grep -q :$next_port — it only sees listening sockets and runs ~6 s before pytest actually binds, so it is inherently racy. The test then constructs dist.TCPStore(..., is_master=True, timeout=timedelta(seconds=5)) and the client-side connect/validate to its own store on 127.0.0.1:10507 never completed inside that 5 s window. The same log shows this container/node was under heavy load and network-stack pressure at that moment: etcd logged put ... took too long (242.697814ms), and UCX logged connect(fd=43, dest_addr=100.107.4.249:52615) failed: Connection refused plus the somaxconn / tcp_max_syn_backlog / UCX_TCP_MAX_CONN_RETRIES hint. A 5 s hard timeout on a fixed, pre-reserved port is the only thing in the Python suite that depends on this; every other test/example uses an OS-assigned port or etcd.

Implicated commit: aed5ef2 — "test: add Python TCPStore metadata integration (#2148)", aschwartz12 (added both the test and the NIXL_TCPSTORE_PORT plumbing in .gitlab/test_python.sh)

File: test/python/test_tcpstore_metadata.py:21-28 (and .ci/scripts/common.sh:38-61, .gitlab/test_python.sh:70)

Suggested fix:

1. Stop depending on the preassigned port — call dist.TCPStore(host_name="127.0.0.1", port=0, ...) in CI too; the test already reads back tcp_store.port on line 33, so nothing else changes. This removes the race with get_next_tcp_port entirely, and NIXL_TCPSTORE_PORT can be dropped from .gitlab/test_python.sh.
2. Raise the store timeout from 5 s to something loaded-CI-safe (e.g. 60 s, still well inside the @pytest.mark.timeout(20)… so bump that marker accordingly, e.g. 90 s) and/or wrap store construction in a small retry loop.
3. Separately, harden get_next_tcp_port: ss -tuln misses non-listening sockets, so prefer actually binding a probe socket (or ss -tan state all) if fixed ports must be reserved.

Related: #2148 (added the test); #1685 (previous port-pool fix in .ci/scripts/common.sh)