CI: Increase cleanup job timeout to 180 minutes

[ntsemah]

What?

Increase cleanup job timeout to 180 minutes

Why?

Job runs daily and currently takes ~55 min. Raised the
limit to handle growth in artifacts so the job does not
time out before finishing cleanup.

How?

It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.

Summary by CodeRabbit

• Chores
  • Extended the cleanup job timeout to improve reliability when removing build artifacts.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi ntsemah! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[ntsemah]

/build

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: eba828b8-4467-4c31-8d43-87a595b3a1d1

📥 Commits

Reviewing files that changed from the base of the PR and between a311600 and 5f5539c.

📒 Files selected for processing (1)

• .ci/jenkins/lib/cleanup-matrix.yaml

---

📝 Walkthrough

Walkthrough

The Jenkins cleanup matrix configuration increases the nixl-ci-cleanup-artifacts job timeout from 60 to 180 minutes.

Changes

Cleanup job configuration

Layer / File(s)	Summary
Artifact cleanup timeout .ci/jenkins/lib/cleanup-matrix.yaml	The nixl-ci-cleanup-artifacts job timeout is increased from 60 to 180 minutes.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: increasing the cleanup job timeout to 180 minutes.
Description check	✅ Passed	The description includes the required What and Why sections and aligns with the template; How is optional.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[ntsemah]

/ok to test 34ef52a

[ntsemah]

/build

[ntsemah]

/ok to test a311600

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 250cd930

TL;DR: The ucx-v1.22.x nixlbench stage crashed with SIGSEGV (exit 139) in the final UCCL-backend DRAM→DRAM loopback WRITE test during engine teardown; the UCCL engine destructor double-destroys / uses-after-free the shared connection object on a self (local) transfer. Fix the destructor to dedup connection destruction by conn_id (the actual uccl_conn_t*), not by agent_name.

Full analysis

Summary: Stage "Run Nixlbench tests" (build variant ucx-v1.22.x, node 299) failed — nixlbench --backend UCCL --op_type WRITE --initiator_seg_type DRAM --target_seg_type DRAM on loopback exited with code 139 (SIGSEGV) at engine destruction.

Root cause: In nixlUcclEngine::~nixlUcclEngine() the connection-teardown loop dedups by agent_name before calling uccl_engine_conn_destroy(conn). In the single-node loopback DRAM→DRAM case the initiator and target are the same agent, so the same underlying uccl_conn_t* is referenced by multiple connected_agents_ entries (self-connect). The current logic destroys the shared connection and can invoke destroy on an already-freed/aliased connection, causing a use-after-free/double-free segfault at "Destroying Engine". Timestamps show continuous activity up to the crash (no gaps), so this is a real crash, not a hang/timeout. The ucx-master variant passed the identical test, indicating the crash is timing/version-sensitive rather than universal.

Implicated commit: [REDACTED:Hex High Entropy String] "Enable local xfer for UCCL backend (#1428)" — Pravein Govindan Kannan (introduced the local/loopback transfer path that shares a connection with the local agent).

File: src/plugins/uccl/uccl_backend.cpp:42-55 (destructor connection-destroy loop; dedup keyed on agent_name).

Suggested fix: Dedup connection destruction by the actual connection pointer/id rather than the agent name — track destroyed conn_id values in the destroyed_agents/a std::set<uccl_conn_t*> and skip already-destroyed connections; also null out entries after destroy so the loopback self-connection (same conn_id under initiator and target names) is only destroyed once. Additionally guard against destroying a connection that equals the engine's own local/self connection.

Related: PRs #1428 (local xfer for UCCL), #1151 (UCCL consistency checks), #895 (initial UCCL backend). No existing issue found for this specific segfault.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ad360f79-85ab-4836-90d5-b485dc9c1f21 in the triage console for the audit trail.

[ntsemah]

/build

[ntsemah]

/ok to test 5f5539c

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 71a8325b

TL;DR: The v1.22.x "Run Nixlbench tests" stage was aborted (exit 143) after hanging ~18 minutes during shutdown of a UCCL-backend ETCD benchmark; the fix is to debug the UCCL engine/RDMA connection teardown hang — not to raise the cleanup timeout as PR #1964 proposes.

Full analysis

Summary: nixl-ci-gpu #2843 build helper v1.22.x "Run Nixlbench tests" (node 356) was SIGTERM-killed (exit 143) by Jenkins after a wall-clock timeout.

Root cause: A hang, not a slow test. The log shows continuous per-iteration output up to 2026-07-20T09:02:17.159, whose last lines are Engine destroyed and [WARN … event_loop rdma/epoll_client.h:207] Error/HUP on connection: 1.1.101.1:33795 — the end of the UCCL-backend / ETCD-runtime / op_type=READ initiator=VRAM target=DRAM iteration. After that there is ~18 minutes of total silence (the single largest gap and the dominant portion of runtime) until Jenkins sends the interrupt at 09:20:01. The next matrix iteration (UCCL READ DRAM→VRAM) never printed even its shell-trace line, so the just-finished UCCL worker process hung during engine/RDMA connection teardown after the epoll HUP event. PR #1964 ("Increase cleanup job timeout to 180 minutes") only masks this hang rather than fixing it.

Implicated commit: unknown for the hang itself; the UCCL teardown path was last substantially touched by [REDACTED:Hex High Entropy String] (Pravein Govindan Kannan, "Enable local xfer for UCCL backend", #1428) and [REDACTED:Hex High Entropy String] ("UCCL: Simplify and Optimize for batch transfers", #1271). The PR under test, #1964, is a timeout increase and does not address the root cause.

File: UCCL RDMA epoll client event loop — reported as rdma/epoll_client.h:207 (the event_loop "Error/HUP on connection" handler) in the UCCL plugin; teardown invoked from the "Destroying Engine" path. (Exact path not resolvable via src/plugins/uccl/rdma/epoll_client.h; it is bundled from the UCCL dependency.)

Suggested fix: Investigate the UCCL engine shutdown path that runs after an RDMA connection HUP. The worker completes the transfer, prints Destroying Engine…/Engine destroyed, then blocks indefinitely — likely a join/wait on the epoll/progress thread or an unsignaled condition when the peer connection is dropped (HUP). Add a bounded timeout/join with deadline in the engine destructor and ensure the HUP handler wakes any threads blocked in the event loop. Do not merge PR #1964's timeout bump as the resolution; at most keep it as a mitigation while the teardown deadlock is fixed. Reproduce locally with the exact failing config: nixlbench --backend UCCL --runtime_type=ETCD --op_type READ --initiator_seg_type VRAM --target_seg_type DRAM.

Related: PR #1964 (the timeout-increase change under test) — #1964 ; UCCL history PRs #1428, #1271, #895.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 7ac7efa9-b030-4d71-a528-e3e847c5db44 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 71a8325b

TL;DR: The Run CPP tests stage aborted (exit 134 / SIGABRT) because ./bin/nixl_test initiator threw an uncaught std::runtime_error when its metadata listener failed to bind port 25203 with EADDRINUSE — the same port was handed to both the target and initiator processes, and it collided (with the peer's listener and/or a lingering socket from the preceding gtest run). This is a CI port-allocation race, not a defect in PR #1964; retry, and harden the port selection / MD-bind handling.

Full analysis

Summary: nixl_test initiator crashed with SIGABRT (Failed to bind metadata listener socket, Address already in use [98]) during the final client/server test of the CPP test stage.

Root cause: In .ci/scripts/test_cpp.sh the client-server test derives one port via get_next_tcp_port() and passes that single port to both nixl_test target and nixl_test initiator. Each instance opens its own MD listener on that port, so the initiator's bind collides with EADDRINUSE. get_next_tcp_port()'s ss -tuln guard only skips ports already in LISTEN state before launch; it cannot prevent this same-port target/initiator collision, nor a TIME_WAIT socket lingering from the immediately preceding gtest-parallel run (which used the adjacent gtest port band). The uncaught exception in metadata_stream.cpp:83 aborts the process (exit 134), and parallel --halt now,fail=1 fails the stage. The log shows continuous activity up to the crash (no multi-minute gaps), so this is a hard bind failure, not a hang or a timeout.

Implicated commit: Not a code regression from commit 71a8325 / PR #1964 (unrelated); the port-allocation logic dates to [REDACTED:Hex High Entropy String] (Alexey Rivkin, "CI: cap port pool below kernel ephemeral range", #1685) and earlier. Root cause is in CI infrastructure, not the PR under test.

File: .ci/scripts/test_cpp.sh:118-119 (single port shared by target+initiator) and .ci/scripts/common.sh:38-61 (get_next_tcp_port race) — surfaced at metadata_stream.cpp:83 (throws std::runtime_error on bind failure).

Suggested fix: First, re-run the build — this is a flaky port collision and the parallel ucx-v1.22.x lane passed. To fix properly: (1) make the MD-listener bind in metadata_stream.cpp retry on EADDRINUSE (and set SO_REUSEADDR to tolerate TIME_WAIT) instead of throwing immediately; and/or (2) in test_cpp.sh, give the initiator a distinct listener port (don't reuse the target's port), and have get_next_tcp_port() verify the port is free by actually attempting a bind rather than relying on ss -tuln, which misses TIME_WAIT/ephemeral sockets from the preceding gtest run.

Related: #1685 (port pool capping), #1081/#1294 (prior CPP test reliability tweaks); no existing issue matches this exact bind race — worth filing one.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id f0a8f4e1-2b72-42cf-8a2e-475d15bf6721 in the triage console for the audit trail.