telemetry: don't unlink the producer's file when attaching fails

[EylonKrause]

What?

sharedRingBuffer<T>::openCyclicBuffer() (src/utils/common/cyclic_buffer.tpp)
is the reader/attach path (create == false), but on two error branches it
unlink()s the shared-memory file it only opened:

• header mmap failure (line 211)
• version mismatch (line 223)

This PR removes those two unlink() calls.

Why?

A reader is attaching to a file that a producer agent created and is actively
using. Deleting it on the reader's error path removes the producer's live
telemetry file from the filesystem. The most realistic trigger is a version
mismatch: a reader built from a different NIXL version (or attaching to an
exporter file from an older run) reads version != TELEMETRY_VERSION and unlinks
the producer's file.

The reader's other two error paths already do the right thing — "File too small
for buffer data" (~line 236) and the final whole-buffer mmap failure (~line 247)
both only munmap and throw, without unlinking. So this just makes all of
openCyclicBuffer consistent: a reader never removes a file it did not create.
The unlink()s in createCyclicBuffer() (the create == true path) are correct
and left unchanged — a creator may remove a file it just made. The file_fd
unique-ptr still closes the descriptor on the error path.

Reproduction

A self-contained fs+mmap reproducer: a "producer" creates the file stamped with an
older version, and a "reader" attaches expecting a newer version.

BEFORE:  file present before reader: yes -> version mismatch -> present after: NO   (reader deleted it)
AFTER :  file present before reader: yes -> version mismatch -> present after: yes  (intact)

How (verification)

• Confirmed the before/after with the reproducer above.
• Compiled the telemetry consumers (buffer_exporter.cpp, telemetry.cpp — which
  instantiate sharedRingBuffer / include cyclic_buffer.tpp) in-tree with
  -Dsanitizer=address,undefined (exit 0).

Happy to add a GoogleTest regression (construct the reader with a mismatched
version under EXPECT_THROW, then assert the file still exists) in test/gtest/
if you'd like one.

Related Issues

None.

Summary by CodeRabbit

• Bug Fixes
  • Improved error handling when opening cyclic buffers: on header mapping failures or on-disk version mismatches, the backing file is now preserved instead of being removed.
  • Errors are still logged and reported, but the existing file remains available for inspection and recovery.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi EylonKrause! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

📝 Walkthrough

Walkthrough

openCyclicBuffer no longer removes the backing file on header mmap failure or version mismatch. Both branches still log the error and throw std::runtime_error.

Changes

Preserve backing file on open errors

Layer / File(s)	Summary
Remove unlink on open failure paths src/utils/common/cyclic_buffer.tpp	Updates the file header comment and removes unlink(name.c_str()) from the header mmap failure branch and the version-mismatch branch in openCyclicBuffer.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐇 Two unlinks hopped out of sight,
The buffer’s file stayed put tonight.
On mmap woes and version fuss,
It logs and throws, but keeps the husk.
A gentler hop for error-light.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: stopping reader-side unlinking when attaching fails.
Description check	✅ Passed	The description follows the template well with What, Why, How, and Related Issues sections and includes verification details.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/utils/common/cyclic_buffer.tpp (1)

221-224: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Update telemetry docs to match new mismatch handling.

The C++ reader no longer unlinks on version mismatch, but docs/telemetry.md still documents unlink-on-mismatch behavior. Please update the docs contract to avoid operator confusion.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/common/cyclic_buffer.tpp` around lines 221 - 224, The
version-mismatch handling in cyclic_buffer no longer unlinks the buffer, so the
telemetry documentation contract is now outdated. Update docs/telemetry.md to
reflect the current behavior described by the mismatch path in
cyclic_buffer.tpp, using the existing version-mismatch handling and NIXL_ERROR
semantics as the source of truth, and remove any mention that the reader unlinks
on mismatch.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/utils/common/cyclic_buffer.tpp`:
- Around line 221-224: The version-mismatch handling in cyclic_buffer no longer
unlinks the buffer, so the telemetry documentation contract is now outdated.
Update docs/telemetry.md to reflect the current behavior described by the
mismatch path in cyclic_buffer.tpp, using the existing version-mismatch handling
and NIXL_ERROR semantics as the source of truth, and remove any mention that the
reader unlinks on mismatch.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 97832167-6985-4419-b9fa-14521125d7fd

📥 Commits

Reviewing files that changed from the base of the PR and between 5047ba2 and c2779a7.

📒 Files selected for processing (1)

• src/utils/common/cyclic_buffer.tpp

[iyastreb]

/build

[EylonKrause]

Disclosure: this contribution was authored with an AI coding assistant (Claude) and reviewed before submission.

[iyastreb]

/build

[iyastreb]

/build

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2608…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit a18383a0

I have sufficient evidence for a confident diagnosis. Let me summarize.

Diagnosis

Timeline analysis of the log (timestamps show continuous activity, no multi-minute gaps):

• 07:24:09 — UCCL WRITE DRAM→DRAM (ASIO runtime) test starts; both workers reach "All processes are ready", connect, and complete the benchmark line (2.317336 GB/s).
• 07:24:11 — Both workers print Destroying Engine... and Engine destroyed.
• 07:24:12 — GNU parallel reports the job failed; srun: error: mizu01: task 0: Exited with exit code 139.

Exit code 139 = 128 + 11 = SIGSEGV. This is a crash during shutdown, not a timeout — the process finished the transfer and printed its result, then segfaulted while tearing down the engine. This is confirmed by the ulimit -c unlimited / "see docs/DebugCoreDumps.md" in the harness (a core dump was collected).

Why it's intermittent / a race: The identical configuration (UCCL WRITE DRAM→DRAM) later succeeded at 07:29:10 under the ETCD runtime. The crash only occurred once, at engine destruction. In nixlUcclEngine::~nixlUcclEngine() (uccl_backend.cpp), the destructor sets stop_listener_ = true, calls uccl_engine_stop_accept(), joins the listener thread, then destroys MR/connection state and finally uccl_engine_destroy(engine_). The startListener() thread loops on uccl_engine_accept() and, on this run, was still mid-accept during teardown (note the Stop background accept... lines seen elsewhere are absent from this particular failing worker's teardown). A connection object being finalized concurrently with the listener/local-xfer path is the classic use-after-free / double-free that produces SIGSEGV at shutdown.

The local same-node transfer path (added in #1428, "Enable local xfer for UCCL backend", [REDACTED:Hex High Entropy String]) plus the connection-teardown ordering in the destructor is the implicated area.

---

Summary: The nixlbench --backend UCCL --op_type WRITE --initiator_seg_type DRAM --target_seg_type DRAM (ASIO runtime) worker segfaulted (exit 139/SIGSEGV) during engine shutdown, failing the "Run Nixlbench tests" stage.

Root cause: Not a timeout — a crash on shutdown. The UCCL engine finished the transfer (result line printed, Destroying Engine... logged) then SIGSEGV'd during ~nixlUcclEngine(). Evidence points to a shutdown/teardown race between the startListener() accept loop and connection/MR destruction in the destructor, exercised on the same-node local-transfer path. It is intermittent — the same config succeeded later under the ETCD runtime.

Implicated commit: [REDACTED:Hex High Entropy String] "Enable local xfer for UCCL backend (#1428)" (Pravein Govindan Kannan) — introduced/expanded the local-xfer and connection-lifecycle handling. (Definitive attribution requires the collected core dump backtrace.)

File: src/plugins/uccl/uccl_backend.cpp:80-120 (~nixlUcclEngine() teardown ordering vs. startListener() accept loop).

Suggested fix:

1. Retrieve the core dump (harness saved one; see docs/DebugCoreDumps.md) and get the backtrace to pin the exact faulting frame.
2. In the destructor, ensure the listener thread is fully stopped and joined before any connection/MR objects it may reference are destroyed, and guard connected_agents_/mem_reg_info_ access shared with the accept/local-xfer path under a single consistent lock ordering. Confirm uccl_engine_stop_accept() actually unblocks a pending uccl_engine_accept() (avoid the thread being mid-accept on a freed connection).
3. Add a null/already-destroyed guard on connection pointers to prevent double-free.

Related: PR #1428 (local xfer for UCCL); no existing issue found for this exact SIGSEGV — recommend filing one and attaching the backtrace.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8266fa93-ca31-4ce0-8215-8611f3419102 in the triage console for the audit trail.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2611…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit a18383a0

This is the UCCL plugin source. The crash is in the UCCL backend runtime path (a native SIGSEGV inside nixlbench using UCCL). Importantly, this is an intermittent/pre-existing crash in the UCCL backend, unrelated to PR #1826 (a telemetry-only change).

Let me confirm the crash context by examining the timeline. The key evidence in the log:

• At 08:31:10.972: parallel: This job failed: ./bin/nixlbench ... --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type DRAM
• At 08:31:12.870: srun: error: mizu01: task 0: Exited with exit code 139

Exit 139 = SIGSEGV. The process crashed — there is continuous activity right up to the crash (transfers completing every few seconds, no multi-minute silent gap), so this is not a hang/timeout. It's a hard segfault in the UCCL WRITE VRAM→DRAM path on the cross-node (ETCD) run on mizu01.

I have enough evidence for a confident diagnosis.

---

Summary: The nixl-ci-gpu #2611 GPU benchmark stage failed because a nixlbench process crashed with a segmentation fault (exit code 139 / SIGSEGV) during the UCCL-backend WRITE test with initiator_seg_type=VRAM, target_seg_type=DRAM (cross-node ETCD run on mizu01).

Root cause: A native SIGSEGV inside the UCCL backend transfer path, not a timeout or hang. The log shows continuous benchmark activity right up to the crash (each transfer completing within seconds; no multi-minute silent gap), so the job was making progress and then hard-crashed. GNU parallel was invoked with --halt now,fail=1, so this single crash aborted the whole stage:

• 08:31:10.972 parallel: This job failed: ./bin/nixlbench … --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type DRAM --check_consistency
• 08:31:12.870 srun: error: mizu01: task 0: Exited with exit code 139
• ERROR: parallel task failed … script returned exit code 139

This is in the UCCL backend/nixlbench worker — the crash is independent of the PR under test. PR #1826 ("telemetry: don't unlink the producer's file when attaching fails") only touches telemetry code and cannot produce a segfault in the UCCL WRITE path. The many [ERROR] Failed to set thread affinity to NUMA node 1 lines are pre-existing benign warnings seen throughout all UCCL runs (including passing ones), not the cause. The crash appears to be a flaky/intermittent UCCL-backend fault: numerous other UCCL VRAM/DRAM WRITE/READ combinations succeeded in the same run, and only this one instance segfaulted.

Implicated commit: unknown — not attributable to the tested commit a18383a0 (PR #1826 is telemetry-only). Crash originates in the UCCL backend, last substantively changed by [REDACTED:Hex High Entropy String] (Pravein Govindan Kannan, "Enable local xfer for UCCL backend", #1428) and [REDACTED:Hex High Entropy String] (Colin Hirsch, "BACKENDS: Unified get param.", #1595); no UCCL change lines up with this build.

File: src/plugins/uccl/uccl_backend.cpp (UCCL WRITE transfer path exercised by nixlbench --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type DRAM); exact faulting line not available without a core dump/stack trace (the repo notes docs/DebugCoreDumps.md).

Suggested fix:

1. Re-run the job to confirm this is the intermittent UCCL crash and not deterministic (it is very likely unrelated to PR telemetry: don't unlink the producer's file when attaching fails #1826, so a retry should be attempted before blaming the PR).
2. Capture the core dump / backtrace per docs/DebugCoreDumps.md for the nixlbench --backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type DRAM invocation to localize the faulting frame in the UCCL backend.
3. Open/attach to a UCCL-backend flakiness tracking issue; consider not gating the whole PR CI on the UCCL WRITE VRAM→DRAM case (or marking it allowed-to-retry) until the segfault is root-caused, so unrelated PRs like telemetry: don't unlink the producer's file when attaching fails #1826 aren't blocked.

Related: PR #1826 (the triggering PR — telemetry only, not the cause); UCCL history PRs #1428, #1595, #1271, #1151, #895. No existing issue specifically matches this segfault signature.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1163759d-cbea-4f90-9d81-5abb94d701a0 in the triage console for the audit trail.

[iyastreb]

/build

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2625…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 3ca78ebe

I have the key evidence. Let me analyze the two failing stages.

Stage 124 (build_helper/x86_64/ucx-master): This is the "Compiling NIXL Docker Image" step for the ucx-master variant. The log shows the entire build actually succeeded — UCX master built and installed, NIXL built and installed, nixlbench built and installed, the image was tagged and pushed successfully. The failure came after the image push, in the environment allocation:

salloc: Pending job allocation 62145
salloc: job 62145 queued and waiting for resources
salloc: error: Unable to allocate resources: Connection timed out
script returned exit code 1

Stage 200 (Allocate Environment): Same signature, and this is the real failing stage (~3610s duration ≈ 60 minutes).

Let me examine the timestamps for the hang analysis:

• 13:59:48.948Z — Calling slurm.allocation ... immediateTimeout:3600
• 14:59:58.481Z — INFO: scctl version (next meaningful line)

That is a ~60-minute gap with zero output between the slurm allocation call and the next line. The immediateTimeout:3600 (3600 seconds = 60 minutes) matches exactly. The Slurm scheduler could not allocate the requested GPU node (--gres=gpu:4 --mincpus=24 --mem=128G on partition mizu) within the immediate window and returned "Unable to allocate resources: Connection timed out."

This is not a code/build failure — the entire NIXL + UCX build compiled, linked, and installed cleanly. This is an infrastructure/resource-availability failure: the SLURM cluster had no free GPU node matching the request (or the scheduler control connection timed out) during the 1-hour immediate-allocation window.

Note the largest gap here is a legitimate scheduler queue wait, not a hung application process — the job was queued and waiting for resources the entire time, which is the scheduler doing its job, then timing out because no node became free. This is a capacity/connectivity issue, not a NIXL regression.

Summary: The nixl-ci-gpu build failed in the Allocate Environment stage (and the parallel ucx-master branch's allocation) — not in any compile/test step.

Root cause: SLURM could not allocate the requested GPU node within the 3600s immediate-allocation window on partition mizu. The log shows salloc: job 62145 queued and waiting for resources followed ~60 minutes later by salloc: error: Unable to allocate resources: Connection timed out, and script returned exit code 1. The Docker image build (UCX master + NIXL + nixlbench) fully succeeded and was pushed before this — so this is a cluster resource/connectivity failure, not a code defect in PR #1826.

Implicated commit: unknown — not code-related. The failure is in CI infrastructure (SLURM/scctl allocation), independent of commit 3ca78eb.

File: CI pipeline SLURM allocation step — slurm.allocation(partition: mizu, gres=gpu:4, mincpus=24, mem=128G, immediateTimeout:3600) (Jenkinsfile / shared library swx-jenkins-lib). No repository source file is implicated.

Suggested fix:

• Re-run the build (retry) — the requested GPU node on mizu was simply unavailable/busy during the immediate window. This is a transient capacity failure.
• If it recurs, check mizu partition health and scheduler connectivity (salloc: Unable to allocate resources: Connection timed out can also indicate the scctl/Slurm control endpoint was unreachable, not just a full queue).
• Consider making the allocation step retry with backoff, and/or raise immediateTimeout only if the cluster is known to be legitimately busy (queueing), not if the controller connection itself is timing out.
• Do not treat this as a PR-1826 regression — no test or compile step failed.

Related: none found (infrastructure allocation timeout; not tied to a tracked issue/PR).

[iyastreb]

/build

[iyastreb]

/build

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu-ep build 196…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit abbf906d

The recent CI history confirms this is an actively-evolving DL EP pipeline (added in #1802). The failure has nothing to do with the PR's code changes.

Diagnosis

Summary: The nixl-ci-dl-gpu-ep build #196 failed in the Allocate DL EP Environment stage (node 200); the Slurm salloc request for a GB200 node on dlcluster.nvidia.com timed out after sitting in the queue for the full hour, returning "Unable to allocate resources: Connection timed out."

Root cause: GPU cluster capacity/scheduling, not a NIXL code defect. The Docker image compile (stage 159) and NIXL/UCX/nixlbench builds all completed successfully. The terminal failure is:

• 04:32:19 — ssh … salloc -N 1 -p gb200nvl72_cx8 … --immediate=3600 --time=01:30:00 … issued
• 05:32:28 — ~3600s (1 hour) later, Slurm reports job 1504481 queued and waiting for resources then salloc: error: Unable to allocate resources: Connection timed out

The single ~1-hour gap is the --immediate=3600 window: Slurm waited the maximum allowed time for a free node in the gb200nvl72_cx8 partition, none became available, and it aborted (script exit code 1). This is a scheduler/resource-availability timeout — the process was waiting on the cluster queue, not hung in application code, so this is a legitimate resource-starvation timeout rather than a code hang. (Stage 124's 5.6s "failure" is the known build_helper "pipeline stop"/retry, which succeeded on retry as stage 142 — not the cause.)

Implicated commit: None. PR #1826 (abbf906d) is unrelated; the failure is external infrastructure. Closest relevant CI change is 55bcf4dcfd9f (lishapira, "ci: Add nixl ep to new dl ci pipeline" #1802), which introduced this allocation stage, but it is not the cause.

File: Slurm allocation invocation in the Allocate DL EP Environment stage (slurm.allocation in the shared swx-jenkins-lib; partition gb200nvl72_cx8, --immediate=3600). No NIXL source file is implicated.

Suggested fix:

• Re-run the build — this is a transient capacity failure on the gb200nvl72_cx8 partition; it will likely pass when a GB200 node frees up.
• If these allocation timeouts recur, the CI/infra owners should either (a) raise --immediate / add scheduler retry-with-backoff around the salloc so a queued job isn't hard-failed at exactly 1 hour, (b) check GB200 partition health/quota under the blackwell account, or (c) surface a clearer "resource unavailable" status so the triage agent (and humans) don't treat it as a code failure.
• Do not treat this as a defect in PR telemetry: don't unlink the producer's file when attaching fails #1826; no code change is warranted.

Related: PR #1802 (added the DL EP pipeline / allocation stage), PR #1889 (ci: version the dispatcher pipeline and drop flaky uploadLogs) — both indicate ongoing flakiness work in this pipeline. No matching issue found for the specific salloc … Connection timed out.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2640…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit abbf906d

The exact telemetry path isn't confirmed and I don't want to guess at source I can't read. Let me step back and focus on what the log definitively tells me, since I already have a strong log-based diagnosis.

The critical facts from the log:

• Every UCCL ASIO test ran successfully and printed benchmark rows.
• The final UCCL test (WRITE/DRAM/DRAM) printed its benchmark result and one "Engine destroyed" at 05:02:25.369.
• Then complete silence for ~47 minutes until Jenkins sent SIGTERM at 05:49:43 (exit code 143).
• The very next expected output — start_etcd_server from line 112 — never appeared.

This means the parallel --halt now,fail=1 invocation on line 84/99 for the last UCCL test never returned: one of the two nixlbench UCCL worker processes hung during shutdown (only one "Engine destroyed" was printed by the pair at the end, whereas prior iterations printed two). parallel blocks waiting for both jobs to exit, so the shell script hung, and Jenkins eventually killed it.

This is a hang in nixlbench UCCL worker shutdown, not a slow test and not a timeout to be raised.

Here is my diagnosis:

Summary: The "Run Nixlbench tests" stage (v1.22.x pipeline, node 318) was ABORTED after Jenkins killed it with SIGTERM (exit code 143) following a ~47-minute hang.

Root cause: A hang, not a slow test. The test script iterates through backend/op_type/seg_type combinations. All UCCL ASIO benchmarks produced results; the last one (--backend UCCL --op_type WRITE --initiator_seg_type DRAM --target_seg_type DRAM) printed its result and began teardown at 05:02:24–25, emitting "Destroying Engine…" for both workers but only a single final "Engine destroyed" (05:02:25.369) plus an Error/HUP on connection warning. After that the log is completely silent for ~47 minutes until Jenkins sent the interrupt at 05:49:43. The next expected line — start_etcd_server (test_nixlbench.sh:112) — never printed. This means the parallel --line-buffer --halt now,fail=1 call (test_nixlbench.sh:84, invoked from the UCCL loop at line 99) never returned because one of the two UCCL nixlbench worker processes hung during engine/connection shutdown and never exited, and parallel blocks until both jobs terminate. The UCCL teardown path (Engine destroy / RDMA epoll connection close, seen as epoll_client.h:207 Error/HUP on connection) is where execution stalled.

Implicated commit: unknown — the UCCL backend code (src/plugins/uccl, last real change [REDACTED:Hex High Entropy String] / [REDACTED:Hex High Entropy String]) was not modified by this PR. This build is for PR #1826 (telemetry), which is unrelated to the UCCL shutdown path, so this looks like a pre-existing intermittent UCCL shutdown hang rather than a regression from #1826. (Note also the pin_thread_to_numa … Failed to set thread affinity to NUMA node 1 errors on every UCCL run — an environment/affinity issue worth noting but not the hang cause.)

File: .gitlab/test_nixlbench.sh:99 (UCCL ASIO loop) → hang inside the nixlbench UCCL worker shutdown reached via parallel at .gitlab/test_nixlbench.sh:84; UCCL teardown in the uccl plugin (Engine destroy / rdma/epoll_client.h:207).

Suggested fix:

1. Do not raise the Jenkins time limit — this is a hang, not a slow test.
2. Make the test harness resilient so a stuck worker can't burn the whole allocation: wrap each nixlbench invocation in a timeout (e.g. timeout 120 ./bin/nixlbench … inside run_nixlbench_two_workers_asio) so a hung UCCL shutdown fails fast with a clear error instead of hanging for 47 minutes.
3. Investigate the UCCL backend shutdown path for a deadlock/blocking join: the final iteration produced only one "Engine destroyed" and an Error/HUP on the epoll connection, suggesting the engine-destroy / epoll event-loop teardown can block waiting on a peer that already closed. Reproduce by running the UCCL WRITE DRAM DRAM ASIO two-worker case in a loop and capture a stack trace (gdb / py-spy on the stuck PID) to confirm where it blocks.
4. Since PR telemetry: don't unlink the producer's file when attaching fails #1826 only touches telemetry, re-run the job to confirm this is the known intermittent UCCL shutdown hang and not caused by the PR before attributing it to telemetry: don't unlink the producer's file when attaching fails #1826.

Related: PR #1826 (the PR under test — telemetry, unrelated to the hang); PR #1724 "UCCL crash Fix + GDB trap"; commit 1cca7d3 (#1560, "Add sleep to prevent both nixlbench processes becoming clients") — prior UCCL/nixlbench process-coordination fix in the same test path.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f09dafb6-cac7-4011-9ab5-132dcef92686 in the triage console for the audit trail.

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2647…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit abbf906d

I have enough evidence. Both failing stages (125 for ucx-v1.22.x and 150 for the corresponding Docker image compile) fail identically: the NIXL build itself succeeds completely, and the failure occurs during the final podman push of the built image to the artifactory registry.

Let me note the key evidence and check timestamps around the failure.

The critical lines:

• Build + install of NIXL, UCX, uccl, nixlbench all complete successfully.
• Image is committed and tagged successfully at 10:02:10.
• podman push begins at 10:02:12, copies blobs until ~10:02:20.
• Then at 10:05:27 — after ~3 minutes on the last blob — it fails with:
  Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error
• Compiling NIXL Docker Image failed ... exit code=125

This is a registry-side infrastructure failure, not a code or test failure. The 500 comes from the artifactory server while receiving a chunked layer upload. Note stage 124 (ucx-master variant) succeeded and stage 125 (ucx-v1.22.x variant) failed — same code, different registry outcome, which is characteristic of a flaky/overloaded registry rather than a code defect.

Summary: The nixl-ci-gpu build failed in the "Compiling NIXL Docker Image" stage (v1.22.x variant, node 150; and its build_helper counterpart node 125) during the final podman push to artifactory — not during compilation.

Root cause: Registry-side infrastructure error. The full NIXL/UCX/nixlbench compile and install succeeded and the image was tagged successfully. The failure is Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error from artifactory.nvidia.com while uploading an image layer, causing podman to exit 125. The push began at 10:02:12, stalled ~3 minutes on one blob, then the server returned HTTP 500 at 10:05:27 — an artifactory 5xx (transient server error / overload), unrelated to PR #1826's changes. The parallel ucx-master image (node 124/168) built and pushed fine, confirming this is non-deterministic registry flakiness rather than a code defect.

Implicated commit: None — commit [REDACTED:Hex High Entropy String] is not implicated; the failure is in artifact upload infrastructure.

File: Not a source file. Failure at the podman push artifactory.nvidia.com/.../nixl-ci-gpu-test-v1.22.x:2647 step in .ci pipeline (Dockerfile: .ci/dockerfiles/Dockerfile.gpu-test).

Suggested fix: Retry the build — this is a transient artifactory HTTP 500. To harden the pipeline against recurrence, wrap the podman push in a retry loop with backoff (e.g., 3 attempts) and treat 5xx registry responses as retryable, and/or raise it with the artifactory/infra team if the 500s persist (check registry health/quota around 2026-07-09 10:05 UTC). Do not change application code or test timeouts.

Related: none found (no issue/PR search performed; failure is infrastructure, not code).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 31db2b12-130a-4e5c-a6f7-ada826b02898 in the triage console for the audit trail.

[svc-nixl]

👀 Investigating nixl-ci-dl-gpu-ep build 206…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit abbf906d

I have a clear picture now. Both "Allocate DL EP Environment" stages failed identically. Let me confirm the timeline of the allocation failure.

The evidence is clear from the "Allocate DL EP Environment" stage log:

• 10:16:01 — the salloc SSH command was issued to dlcluster.nvidia.com requesting a node on partition gb200nvl72_cx8 with --immediate=3600 (i.e., wait up to 3600s / 1 hour for an immediate allocation).
• 11:16:10 — exactly one hour later, output resumes: salloc: Pending job allocation 1507363 … salloc: job 1507363 queued and waiting for resources … salloc: error: Unable to allocate resources: Connection timed out.

The single ~60-minute gap (10:16 → 11:16) is the --immediate=3600 timeout elapsing while the Slurm job sat queued and waiting for resources. This is not a hang in application/test code — it's Slurm never granting the allocation because no GB200 node became free within the immediate window, and the SSH connection then timed out. The build itself (image compile, UCX build, NIXL build, nixlbench) all completed successfully; the failure is purely at the cluster resource-allocation step.

Summary: The nixl-ci-dl-gpu-ep build failed at the "Allocate DL EP Environment" stage — the Slurm salloc for a GB200 node did not get resources within the immediate timeout and exited with "Unable to allocate resources: Connection timed out."

Root cause: Cluster/infrastructure resource contention, not a code defect. The pipeline requested an exclusive node on partition gb200nvl72_cx8 with salloc --immediate=3600 --time=01:30:00. The job (1507363) was queued and waiting for resources for the full 3600s immediate window; no GB200NVL72 node freed up, so Slurm aborted the allocation and the SSH session timed out. The log shows one contiguous ~60-minute gap (10:16:01 → 11:16:10) that exactly matches the --immediate=3600 window elapsing — this is a legitimate wait for a busy/unavailable partition, not a hung test process. All build stages (Docker image compile, UCCL/UCX build, NIXL + nixlbench meson/ninja build, image push) completed successfully beforehand.

Implicated commit: unknown — not attributable to commit abbf906 / PR #1826; the failure is in cluster scheduling, and the image built cleanly from this commit.

File: The allocation logic lives in the pipeline slurm.allocation step (swx-jenkins-lib), invoked with partition:gb200nvl72_cx8, immediateTimeout:3600, jobTimeout:01:30:00. No repo source file is implicated.

Suggested fix: This is a transient infrastructure failure — retry the build once GB200 nodes are available; it is not caused by PR #1826. To reduce recurrence: (1) verify GB200NVL72 (gb200nvl72_cx8, --account=blackwell) partition capacity/queue depth around the failure window — the exclusive-node request means it waits for a whole free node; (2) consider adding automatic retry/backoff around the slurm.allocation step, or increasing immediateTimeout if an hour is genuinely too short for this partition's queue; (3) if only a subset of GPUs is needed, request a GPU count instead of exclusive mode (the log notes "Setting job to exclusive mode as no gpu count was specified"), which may allocate faster. Do not treat this as a code bug in the PR.

Related: none found.

[iyastreb]

/build