CI: exclude misconfigured Vera Rubin nodes from nixl-ci-gpu-vr

[NirWolfer]

Excludes four vrnvl72 nodes from the nixl-ci-gpu-vr Slurm allocation:

• vr-nvl72-ts3-l06-comp137, comp139, comp141
• vr-nvl72-ts2b-l10-comp102

These nodes are configured differently from the rest of the partition. On them test_prep_mem_view fails deterministically — UCX rejects the remote mem-view with lane not found for element 0 — while the same image and commit pass on the other nodes. In the bring-up runs every failure landed on an affected node and every pass on an unaffected one.

Changes:

• .ci/jenkins/lib/test-vr-matrix.yaml — new SLURM_EXCLUDE env entry, passed as --exclude=${SLURM_EXCLUDE} in the allocation step's extraArgs, the same way test-llm-container-matrix.yaml already does.
• .ci/docs/ci-overview.md — note the exclusion on the nixl-ci-gpu-vr entry.

Summary by CodeRabbit

• CI
  • Vera Rubin GPU tests now exclude four specified Slurm nodes during allocation.
  • Updated the CI overview to clarify that nodes with differing configurations are excluded.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[NirWolfer]

/build

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/nixl/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: 578e73ee-64a1-4a5a-a002-1964e7c2feda

📥 Commits

Reviewing files that changed from the base of the PR and between 27153aa and 1802112.

📒 Files selected for processing (2)

• .ci/docs/ci-overview.md
• .ci/jenkins/lib/test-vr-matrix.yaml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The Vera Rubin GPU test matrix now passes four configured node exclusions to Slurm allocation. The CI overview documents the SLURM_EXCLUDE behavior.

Changes

Vera Rubin GPU node exclusions

Layer / File(s)	Summary
Configure and document Slurm exclusions .ci/jenkins/lib/test-vr-matrix.yaml, .ci/docs/ci-overview.md	The matrix defines four nodes in SLURM_EXCLUDE and passes the value as a Slurm --exclude argument. The CI overview documents the exclusion behavior.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Bug fix

Suggested reviewers: alexey-rivkin

Merge Risk: ⚪ Minimal · up to 18021

The change documents and configures the four node exclusions. No concrete merge-blocking failure is established, though the allocation library’s handling of the argument remains unconfirmed.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: excluding misconfigured Vera Rubin nodes from the CI allocation.
Description check	✅ Passed	The description explains what the PR changes, why the nodes are excluded, and how the exclusion is applied. It provides the affected nodes and the reported failure details. It does not use the templat…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 18021123

TL;DR: The only failure is test/python/test_tcpstore_metadata.py::test_tcpstore_metadata_exchange, which timed out after 5 s connecting to its own PyTorch TCPStore master on the CI-assigned fixed port 10507 — a port-assignment race/stale-state problem in the CI harness, not a code regression from this PR. Fix the test to bind an OS-assigned port (or retry on DistNetworkError) instead of trusting a pre-allocated NIXL_TCPSTORE_PORT.

Full analysis

Summary: Stage 580 "Test Python" (ubuntu22 variant) failed: 1 of 28 pytest tests failed with torch.distributed.DistNetworkError: The client socket has timed out after 5000ms while trying to connect to (127.0.0.1, 10507); the other 5 parallel variants ran the same test successfully.

Root cause: .gitlab/test_python.sh:70 pre-assigns NIXL_TCPSTORE_PORT via get_next_tcp_port() (.ci/scripts/common.sh:38-61), which picks a port by reading a counter file and probing with ss -tuln | grep -q :$next_port. That check is advisory only — the port is never reserved, and ~9 s elapse between allocation (14:19:30) and the test actually binding it (14:19:39). The test then creates the store with is_master=True and a hard 5-second timeout and no retry (test/python/test_tcpstore_metadata.py:21-28), so any contention on 10507 fails the whole suite. Two log details show the container carried over state from an earlier build on the same agent, which makes a stale occupant of that port range plausible: etcd reported the server is already initialized as member before, starting as etcd member... (pre-existing default.etcd data dir), and /tmp/nixl_tcp_port_0 already existed holding 10504 rather than the range base 10500. The triggering commit 1802112367 ("CI: exclude misconfigured Vera Rubin nodes from nixl-ci-gpu-vr", NirWolfer) only edits Jenkins node labels for the nixl-ci-gpu-vr pipeline and cannot affect this test — the failure is unrelated to PR #2339.

Implicated commit: aed5ef2e — aschwartz12, "test: add Python TCPStore metadata integration (#2148)" (introduced both the test and the fixed-port scheme). The triggering commit 1802112367 (NirWolfer) is not implicated.

File: test/python/test_tcpstore_metadata.py:21-28 (with .ci/scripts/common.sh:38-61 and .gitlab/test_python.sh:70)

Suggested fix: Stop depending on a pre-assigned fixed port in this test. Preferred: pass port=0 and read the kernel-assigned port back from tcp_store.port (the test already uses tcp_store.port on line 33, so nothing else needs to change) and drop NIXL_TCPSTORE_PORT from .gitlab/test_python.sh. If the fixed port must stay for executor isolation, wrap the dist.TCPStore(...) construction in a small retry loop over the next few ports on DistNetworkError and raise the timeout above 5 s. Separately, harden get_next_tcp_port() to actually hold the port (bind-and-close with SO_REUSEADDR, or a lock file) rather than relying on a racy ss | grep, and have the Jenkins job clean /tmp/nixl_tcp_port_* and the stale default.etcd directory at stage start so reused workspaces don't inherit counters or data dirs.

Related: PR #2148 (added the test); no existing issue tracks this flake — worth opening one.