test: reduce ucx_backend_test transfer volume

[NirWolfer]

Summary

The TSan leg of nixl-ci-test-sanitizers was being killed at its 40 minute cap while ucx_backend_test was still making steady progress.

In build #649 the stage started at 10:08:23 and was killed at 10:48:22 — exactly 40:00. ucx_backend_test ran from 10:10:16 to the kill (~38 min), printing data-verification output every 8-12s with a largest gap of ~15s, and was cut off mid-WRITE in the last of its four configurations. Not a hang, and not a sanitizer finding — it simply does not fit. Build #632 hit the same wall.

Each configuration moves and verifies desc_cnt * desc_size bytes. At 64 descriptors of 1 MB that is 64 MB per configuration, and under ThreadSanitizer every access on that data path is instrumented.

Changes

• test/unit/plugins/ucx/ucx_backend_test.cpp — drop desc_cnt from 64 to 8 in test_intra_agent_transfer and test_inter_agent_transfer

Why this does not reduce coverage

• The configuration matrix (progress thread on/off x handler reuse on/off) is unchanged, so every code path is still exercised. Dropping configurations would have cost real coverage; reducing the volume of identical work does not.
• 8 descriptors still exercise the multi-descriptor list handling in populateDescs and the descriptor-list transfer paths.
• desc_size deliberately stays at 1 MB, so transfers remain in the same UCX protocol regime — shrinking the payload instead could cross the eager/rendezvous threshold and silently change which transport path is tested.
• ThreadSanitizer reports races from happens-before analysis on the accesses it observes, not from stress. A race on a given code path is flagged regardless of how many times that path repeats, so 8 descriptors detect what 64 would.

Net effect is ~8x less data per configuration, which brings the stage well inside its budget without touching the timeout.

Verification

Confirmation is a green nixl-ci-test-sanitizers run with the tsan leg completing inside 40 minutes. The asan_ubsan leg was already passing and is unaffected in kind, just faster.

Summary by CodeRabbit

• Tests
  • Reduced the number of transfer descriptors used in UCX transfer tests while preserving descriptor sizing and overall transfer behavior.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 808198d6-8a4f-4a83-bd42-8bc8cc4ba271

📥 Commits

Reviewing files that changed from the base of the PR and between e72eb50 and 5056d9b.

📒 Files selected for processing (1)

• test/unit/plugins/ucx/ucx_backend_test.cpp

---

📝 Walkthrough

Walkthrough

This PR changes two UCX unit tests. Both tests now create 8 transfer descriptors instead of 64.

Changes

UCX transfer test updates

Layer / File(s)	Summary
Transfer descriptor count adjustments test/unit/plugins/ucx/ucx_backend_test.cpp	The intra-agent and inter-agent transfer tests now use 8 descriptors instead of 64.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: brminich

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the reduction of UCX backend test transfer volume.
Description check	✅ Passed	The description explains what changed, why it was needed, how coverage is preserved, and how the change will be verified.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[NirWolfer]

/build

[ColinNV]

Should this perhaps have different values for normal and sanitizer runs?

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 17cae61b

TL;DR: The Run Rust tests stage failed because scctl/srun could not stream the command into the enroot/pyxis container — containerd.sock was gone on the compute node. This is a CI infrastructure/node failure, not a code defect; retry the build (or drain/replace the affected node).

Full analysis

Summary: Jenkins job nixl-ci-gpu #3067 aborted when the Run Rust tests stage (node 241, ucx-v1.22.x branch) failed immediately on launch; the parallel build was then stopped and SLURM job 75300 killed.

Root cause: Container-runtime / cluster infrastructure failure, not a test or source-code bug. When Jenkins tried to run .gitlab/test_rust.sh via scctl --raw-errors client connect -- srun --jobid=75300 ..., the container-exec streaming layer returned:
error streaming command in container: Internal error occurred: unable to upgrade connection: rpc error: code = Unavailable desc = ... dial unix /var/run/containerd/containerd.sock: connect: no such file or directory. The follow-up exportEnroot step failed with the identical containerd-socket error. The prior CPP tests passed and the Python stage was mid pip-build, so the node/container was healthy moments earlier and then its container runtime became unreachable. Log timestamps show the failure occurred within ~3s of the stage starting (13:17:29→13:17:32) — an immediate connection failure, not a hang or timeout.

Implicated commit: none — the failure is unrelated to commit 17cae61 / PR #2042; it is an environment/node issue on the SLURM compute node running job 75300.

File: Not a source file. Failure surfaces at the CI orchestration layer invoking .gitlab/test_rust.sh via scctl/srun (containerd socket /var/run/containerd/containerd.sock).

Suggested fix: Retry the build — this is a transient infrastructure fault. If it recurs, have infra check the health of the compute node backing SLURM job 75300 (containerd daemon running, /var/run/containerd/containerd.sock present, enroot/pyxis session valid), drain/replace the bad node, and consider adding an automatic retry for stages that fail with the containerd.sock ... no such file or directory streaming error so a dead node doesn't fail an entire PR. No code change to nixl is warranted.

Related: none found.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 17cae61b

TL;DR: Both the "Run vLLM sanity" and "Run SGLang sanity" stages were killed by the ~40-minute wall-clock limit (exit 143), but the real cause is a hang inside the pyxis/enroot container image import on the SLURM GPU node — the test script never ran a single line. This is an infra/registry stall, not a code bug, so the fix is to retry the build and investigate node/registry health, not to raise the timeout.

Full analysis

Summary: The vLLM (stage 585, jobid 1755685) and SGLang (stage 561, jobid 1755684) sanity stages hung and were SIGTERM'd after ~2.4M ms.

Root cause: In both stages the last log line is pyxis: importing docker image: artifactory.nvidia.com#/.../[vllm|sglang]-nixl:1292 at 13:35:09–13:35:10, followed by ~40 minutes of total silence until Jenkins sent the interrupt at 14:14:54–14:14:57 ("Sending interrupt signal to process" → "Terminated" → exit code 143). The gap is the entire runtime, so this is a hang, not a slow test. The .gitlab/test_vllm_sglang_sanity.sh script never produced any output (not even its first echo/set -e line), so the stall is entirely inside pyxis importing the freshly-pushed :1292 container image onto the SLURM node — an image-pull/enroot-import infrastructure stall (both frameworks hung at the identical step at the identical time on the same cluster, strongly indicating a shared registry/network/node problem, not the NIXL wheel or the test logic). The "Build sanity image" stage (496) confirms the image built and was pushed successfully at 13:34:32.

Implicated commit: None in application code — this is an infrastructure hang. (The most recent CI change, ad661a721217 by NirWolfer, #2007, "request a whole node for the vLLM/SGLang sanity allocation," only touched allocation, which succeeded in ~19s, so it is not the cause.)

File: Not a source-file defect; failure is external to the repo at the pyxis import step invoked from .ci/ slurm helper (slurm.run → srun ... --container-image=...:1292). .gitlab/test_vllm_sglang_sanity.sh did not execute.

Suggested fix: Retrigger the build; this is almost certainly a transient pyxis/enroot image-import stall. To make it self-healing, wrap the srun invocation with an import/step timeout (e.g. SLURM --time on the step or a timeout around the pyxis import) so a stuck import fails fast and is retried rather than eating the whole 40-minute budget, and check the target GB200 aarch64 node / artifactory registry for import or network stalls around 13:35 UTC. Do not raise the Jenkins wall-clock limit — the process was hung, not making progress.

Related: #2007 (recent sanity-allocation change, NirWolfer), #1995 (SGLang router port fix) — none causal. No matching pyxis-hang issue found.