CI: Skip UCCL nixlbench tests

[ovidiusm]

What?

Skip UCCL nixlbench test due to frequent hangs, see #1999

Summary by CodeRabbit

• Tests
  • Disabled flaky GPU-based UCCL transfer tests in CI pending an upstream fix.
  • Other UCX tests and supporting test infrastructure remain unchanged.

[github-actions]

👋 Hi ovidiusm! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[ovidiusm]

/build

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 002810f7-2691-4b05-af81-c8152cfd25d3

📥 Commits

Reviewing files that changed from the base of the PR and between 34b5da0 and 2fd77d6.

📒 Files selected for processing (1)

• .gitlab/test_nixlbench.sh

---

📝 Walkthrough

Walkthrough

The GPU-conditional UCCL nixlbench transfer tests using etcd are disabled with explanatory TODO comments. Other UCX tests and etcd server lifecycle handling remain unchanged.

Changes

nixlbench CI coverage

Layer / File(s)	Summary
Disable flaky UCCL transfer tests .gitlab/test_nixlbench.sh	Comments out GPU-conditional UCCL etcd transfer tests across READ/WRITE operations and initiator/target segment combinations, with TODO guidance to re-enable them after the upstream issue is fixed.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: aranadive

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main CI change.
Description check	✅ Passed	The description states the change and references the related issue, which is sufficient for this fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 84372322

TL;DR: The ucx-v1.22.x "Run CPP tests" stage was SIGTERM-killed (exit 143) after the gtest suite blew past its wall-clock budget; the root cause is a severe performance regression in the ucx_threadpool engine where each TestTransfer case takes ~140–184s instead of ~10s. Most likely introduced by the threadpool-engine refactor in #1906; investigate the progress-thread/worker handling for the threadpool path.

Full analysis

Summary: nixl-ci-gpu #2964, stage node 203 (build_helper/x86_64/ucx-v1.22.x/2 → "Run CPP tests") aborted with exit code 143 when gtest was interrupted at test 124/154.

Root cause: Not a wall-clock shortfall — a per-test regression. Comparing durations in the same gtest run: ucx/TestTransfer.* and ucx_no_pt/TestTransfer.* each finish in ~10–48s, but every ucx_threadpool/TestTransfer.* and ucx_threadpool_no_pt/TestTransfer.* case takes ~140,000–184,000 ms (2.5–3 min) each (e.g. remoteMDFromSocket 142s, NotificationOnly 151s, SelfNotification 154s, ListenerCommSize 183s). These threadpool-path transfers are ~15× slower — a stall/hang in progression per transfer, not genuine work — which exhausted the stage's time limit and triggered the SIGTERM at 12:06:58 (last activity was test #124 at 12:06:22, i.e. work was "progressing" only because each test stalls to a timeout). The threadpool engine (nixlUcxThreadPoolEngine) only creates a progress thread when shared workers exist; the recent refactor deriving it from nixlUcxThreadEngine appears to have broken/omitted progression of shared workers on the threadpool path, so transfers fall back to slow completion.

Implicated commit: 209abbe9b98e — Ilia Yastrebov, "Refactoring: derive nixlUcxThreadPoolEngine from nixlUcxThreadEngine (#1906)" (most recent change to the exact code path that regressed).

File: src/plugins/ucx/ucx_backend.h:315-363 (nixlUcxThreadEngine / nixlUcxThreadPoolEngine progress-thread + worker handling) and the corresponding progressLoop/sendXferRange implementations in src/plugins/ucx/ucx_backend.cpp.

Suggested fix: Do not raise the CI time limit. Bisect/verify against #1906: confirm the progress thread is actually created and running for the threadpool engine and that it progresses the shared workers used by threadpool transfers. Compare ucx_threadpool vs ucx transfer timings locally; if the progress thread is missing or idle on the threadpool path, restore per-transfer progression (ensure progressLoop covers shared workers for the derived engine) so ucx_threadpool/TestTransfer.* returns to ~10s. If #1906 bisects clean, next check #1880 ("fix infinite loop dropping pending requests on dedicated-thread exit") and #1846 (forcible EP close).

Related: PR #1906 (implicated refactor); PR #1880, #1846 (secondary threadpool/EP-lifecycle candidates). No existing issue found for this specific slowdown.