PLUGINS/UCX: Init only needed fields in UCP - 1.4.0

[iyastreb]

What?

Backport #2017 to v1.4.0

Why?

UCP SGL PR openucx/ucx#11258 increased the size of
request param, which is zero-initialized in NIXL on each operation. This extra
initialization costs 2% of performance in nixlbench (512Bx64000, 8 threads) on
EOS and DFW.

How?

We need to explicitly set only fields that we populate, no need to zero-init
the rest

[github-actions]

👋 Hi iyastreb! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[iyastreb]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit cbaddcca

TL;DR: The "Run Nixlbench tests" stage was aborted after a ~48-minute hang (silence from 13:08:42 to 13:56:47), caused by the known-flaky UCCL nixlbench transfer test (issue #1999); the fix is to also skip the UCCL ASIO loop, which PR #2000 left enabled while only disabling the UCCL ETCD loop.

Full analysis

Summary: Jenkins stage "Run Nixlbench tests" (node 351) was ABORTED after wall-clock timeout because the first UCCL WRITE ASIO benchmark hung indefinitely.

Root cause: Not a genuine slowness/time-budget issue — it's a hang. The last application output is Engine destroyed at 2026-07-31T13:08:42.271Z after the UCCL --op_type READ --initiator DRAM --target DRAM test. The next line is Sending interrupt signal to process at 13:56:47.749Z — a ~48-minute gap of complete silence. The process that hung is the next iteration of the loop, the first UCCL WRITE test (--backend UCCL --op_type WRITE --initiator_seg_type VRAM --target_seg_type VRAM), which never printed "Creating Engine". This is the exact behavior tracked in open issue #1999 (UCCL nixlbench test hangs). PR #2000 attempted to mitigate flakiness but only commented out the UCCL ETCD test loop (.gitlab/test_nixlbench.sh lines 124-131); the UCCL ASIO test loop (lines 95-103) was left active and still hangs.

Implicated commit: 7a244b980b47 (ovidiusm, 2026-07-28, "CI: Skip UCCL nixlbench tests (#2000)") — incomplete skip; and root behavior tracked by issue #1999. UCCL backend transfer logic touched in [REDACTED:Hex High Entropy String] (Pravein Govindan Kannan, "UCCL: Simplify and Optimize for batch transfers").

File: .gitlab/test_nixlbench.sh:95-103 (the still-active UCCL ASIO loop).

Suggested fix: Disable the UCCL ASIO benchmark loop the same way the ETCD one was disabled — wrap/comment out lines 95-103 with a reference to issue #1999 until the underlying UCCL hang is fixed. As a safety net, also wrap each nixlbench invocation in run_nixlbench_two_workers_asio with a timeout (e.g. timeout 300 …) so a single hung transfer fails fast instead of consuming the whole stage's wall-clock budget. Do not simply raise the stage time limit — the process is hung, not slow. The real backend hang should be investigated under issue #1999 (likely in the UCCL engine connect/transfer path, e.g. rdma/epoll_client.h / UCCL engine lazy-creation code).

Related: Issue #1999 (UCCL nixlbench test hangs); PR #2000 (incomplete skip).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 0fc03d5b-bbe4-4b5f-89e7-4f8331e4e2a9 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit cbaddcca

TL;DR: The Python test stage failed only against the UCX-master build: test_empty_notif_tag's fixture timed out (>5s pytest-timeout) inside agent.createBackend("UCX") because UCX-master backend creation slowed progressively from ~1s to >12s; the identical nixl commit passed against UCX v1.22.x, so this is a UCX-master regression, not a nixl code bug.

Full analysis

Summary: Stage 318 ("Run Python tests", ucx-master variant) failed — test/python/test_nixl_api.py::test_empty_notif_tag errored at fixture setup with Failed: Timeout (>5.0s) from pytest-timeout while creating a UCX backend.

Root cause: UCX backend instantiation degraded progressively during the run under UCX master. Timestamps show each createBackend("UCX") taking ~1s early (19:01:15–19:01:24) but climbing to ~11–12s later (19:01:54, 19:02:05, 19:02:17, 19:02:28, 19:02:40, 19:02:52, 19:03:04, 19:03:16, 19:03:28), eventually breaching the 5s per-test timeout at _api.py:400 (self.agent.createBackend(backend, initParams)). The matching ucx-v1.22.x variant (stage 261) ran the same tests to completion with no slowdown, and the build_helper/x86_64/ucx-master/1 step (stage 124) itself flagged as FAILURE — pointing at UCX master, not nixl. This is a UCX-master regression (worker/backend init slowdown), surfaced as a pytest-timeout symptom.

Implicated commit: Not a nixl commit — the failing behavior tracks the UCX master dependency, not PR #2022's changes ([REDACTED:Hex High Entropy String]). Specific UCX master commit is unknown from these logs.

File: Test surface: test/python/test_nixl_api.py:58 (fixture two_connected_agents) → nixl_cu13/_api.py:400 (createBackend). Underlying cause is in the UCX master library, not nixl source.

Suggested fix: Treat the ucx-master leg as the culprit and confirm by re-running: (1) rerun the build to check if the slowdown is intermittent/environmental; (2) pin/bisect the UCX master revision used by build_helper/x86_64/ucx-master to identify the regressing UCX commit and report upstream; (3) since UCX master is a moving, non-gating dependency, consider making the ucx-master Python leg non-blocking (allow-failure) so a UCX upstream regression doesn't red-flag nixl PRs, while keeping ucx-v1.22.x as the gating variant. Do not simply raise the pytest-timeout — the >12s backend-creation time is abnormal and is the real regression to investigate.

Related: none found (issue/PR search returned no matching reports for this timeout signature).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 49b08abb-b465-4ba8-b222-8e434bcf7e7d in the triage console for the audit trail.