ci: version the dispatcher pipeline and drop flaky uploadLogs

[NirWolfer]

What?

Move the dispatcher logic out of the inline dsl block in proj-jjb.yaml into .ci/jenkins/pipeline/Jenkinsfile.dispatcher, checked out via pipeline-scm. The build is triggered by a /build issue-comment webhook, so the dispatcher is
checked out from the PR merge ref (refs/pull//merge, derived from the PR number in the payload). This runs the PR merged into its base branch, so a PR into a release branch runs that branch's dispatcher and changes on main no
longer disturb it.

Also remove githubHelper.uploadLogs from the main Jenkinsfile: it sometimes
hangs and kills the runner.

Why?

CI improvements for release branches
CI stability for Blossom-CI issues

How?

It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.

Summary by CodeRabbit

Summary by CodeRabbit

• New Features

  • Added a Jenkins CI dispatcher pipeline that fans out selected leaf jobs in parallel and checks that all complete successfully.
  • Updated the dispatcher job template to run the dispatcher script from the pull request merge ref (or a provided sha1) via SCM.

• Bug Fixes

  • The dispatcher now aborts stale in-progress dispatcher runs (and any leaf builds they started) before starting a new dispatch.
  • Streamlined GitHub commit status updates during dispatcher cleanup and start/end reporting.

• Documentation

  • Refined the Jenkins CI overview, including how LEAF_JOBS temporarily restricts which leaf jobs run.

• Chores

  • Stopped uploading pipeline logs to GitHub during cleanup.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[NirWolfer]

/build

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a dispatcher Jenkinsfile, moves dispatcher execution to SCM-based loading, updates CI docs for the new flow, and removes a log-upload call from the main Jenkinsfile cleanup path.

Changes

Jenkins dispatcher pipeline changes

Layer / File(s)	Summary
Dispatcher pipeline: auth, cleanup, fan-out, status updates .ci/jenkins/pipeline/Jenkinsfile.dispatcher	Adds the dispatcher pipeline header, GitHub credential setup, stale dispatcher and leaf build cleanup, Blue Ocean URL and PENDING status updates, the parallel downstream job map, and the final success/failure dispatch path.
proj-jjb.yaml wiring to SCM-based dispatcher execution .ci/jenkins/pipeline/proj-jjb.yaml	Extends webhook extraction with a sha1 field, adds the dispatcher sha1 parameter, switches the dispatcher job to pipeline-scm with a script-path Jenkinsfile, removes the inline dispatcher DSL, and adds the dispatcher Jenkinsfile project variable.
CI docs and Jenkinsfile cleanup .ci/docs/ci-overview.md, .ci/jenkins/pipeline/Jenkinsfile	Updates the CI overview text for dispatcher checkout, stale-run handling, and LEAF_JOBS, and removes the githubHelper.uploadLogs call from the main Jenkinsfile cleanup path while keeping the commit-status update.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Webhook as GitHub webhook
  participant JJB as Jenkins Job Builder
  participant Dispatcher as Jenkins dispatcher pipeline
  participant GitHub as GitHub commit status API
  participant LeafJobs as downstream leaf jobs

  Webhook->>JJB: deliver PR payload
  JJB->>Dispatcher: check out dispatcher Jenkinsfile with sha1
  Dispatcher->>Dispatcher: abort stale dispatcher and leaf builds
  Dispatcher->>GitHub: set NIXL CI started = PENDING
  Dispatcher->>LeafJobs: run selected leaf jobs in parallel
  LeafJobs-->>Dispatcher: collect results
  Dispatcher->>GitHub: set NIXL CI ended = SUCCESS or FAILURE

Loading

Possibly related PRs

• ai-dynamo/nixl#1871: Also updates .ci/docs/ci-overview.md around the Jenkins dispatcher flow and its execution model.

Suggested reviewers: dpressle

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the main changes: dispatcher pipeline extraction/versioning and removal of flaky uploadLogs.
Description check	✅ Passed	The description covers What and Why clearly, and the optional How section may be left blank.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/pipeline/Jenkinsfile.dispatcher:
- Around line 74-110: The parallel branch definitions in Jenkinsfile.dispatcher
are duplicated and each branch repeatedly calls githubHelper.getMergedSHA(), so
hoist that value once before the parallel block and build the branch map from a
shared job list. Keep the existing branch names and job identifiers (non-gpu,
gpu, dl-gpu, dl-gpu-ep, wheel, sanitizers) but refactor the parallel closure
setup so each build step reuses the single merged SHA and the same githubData
payload.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 26e53c2a-320f-406f-be1a-d697107ff4a6

📥 Commits

Reviewing files that changed from the base of the PR and between f948f93 and 3ee94c7.

📒 Files selected for processing (3)

• .ci/jenkins/pipeline/Jenkinsfile
• .ci/jenkins/pipeline/Jenkinsfile.dispatcher
• .ci/jenkins/pipeline/proj-jjb.yaml

💤 Files with no reviewable changes (1)

• .ci/jenkins/pipeline/Jenkinsfile

[NirWolfer]

/build

[NirWolfer]

Copy of the new dispatcher job ran at: https://nbuprod.blsm.nvidia.com/nbu-swx-nixl-main/job/Devops/job/nixl-ci-devops-dispatcher-test/6/

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/pipeline/Jenkinsfile.dispatcher:
- Around line 88-94: The dispatcher currently ignores leaf-job failures because
Jenkinsfile.dispatcher uses build with propagate: false, so unsuccessful child
jobs can still lead to a success path. Update the parallel branch handling
around the build call to capture the returned result for each leaf job and
explicitly throw or fail when it is anything other than SUCCESS, using the
existing leafJobs/branchName build flow to locate the change.

In @.ci/jenkins/pipeline/proj-jjb.yaml:
- Around line 62-70: The trusted dispatcher is currently being loaded from the
PR merge ref in the pipeline SCM checkout, which allows untrusted PR changes to
affect the sandbox:false dispatcher path. Update the SCM/refspec handling in the
Jenkins pipeline config so the privileged dispatcher is fetched from a trusted
base branch or fixed ref, while keeping refs/pull/$sha1/merge only for the leaf
jobs. Use the pipeline-scm and dispatcher-loading logic in this YAML to separate
dispatcher checkout from PR merge checkout.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 92b29e96-814f-4470-8a54-a19b32be328e

📥 Commits

Reviewing files that changed from the base of the PR and between 3ee94c7 and 3625817.

📒 Files selected for processing (3)

• .ci/jenkins/pipeline/Jenkinsfile
• .ci/jenkins/pipeline/Jenkinsfile.dispatcher
• .ci/jenkins/pipeline/proj-jjb.yaml

💤 Files with no reviewable changes (1)

• .ci/jenkins/pipeline/Jenkinsfile

[NirWolfer]

/build

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/pipeline/Jenkinsfile.dispatcher:
- Around line 70-72: The Blue Ocean link in the commit status is built with
env.JOB_BASE_NAME, which strips folder components and breaks foldered jobs.
Update the blueOceanUrl construction in Jenkinsfile.dispatcher to follow the
main Jenkinsfile pattern by using env.JOB_NAME and encoding it when building the
URL, then keep githubHelper.updateCommitStatus using that corrected link.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: e7ef877f-ecb3-4f9e-b216-03da2ede31fa

📥 Commits

Reviewing files that changed from the base of the PR and between 3625817 and 4a09308.

📒 Files selected for processing (3)

• .ci/jenkins/pipeline/Jenkinsfile
• .ci/jenkins/pipeline/Jenkinsfile.dispatcher
• .ci/jenkins/pipeline/proj-jjb.yaml

💤 Files with no reviewable changes (1)

• .ci/jenkins/pipeline/Jenkinsfile

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/pipeline/Jenkinsfile.dispatcher:
- Around line 49-59: The stale leaf-job cleanup in Jenkinsfile.dispatcher only
discovers children by scanning the console log, so queued or not-yet-started
jobs can be missed and keep running after the new build starts. Update the leaf
discovery around the staleBuilds loop to also look up queued items from
Jenkins.instance.queue or otherwise track child runs by upstream cause, so both
running and queued descendants are collected. Keep the existing best-effort
try/catch behavior, and ensure the collected leafBuilds includes these
additional child references before aborting them.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1a8e9ba6-eabd-48e7-9915-d5fc5fe9f553

📥 Commits

Reviewing files that changed from the base of the PR and between 4a09308 and e0a17e5.

📒 Files selected for processing (4)

• .ci/docs/ci-overview.md
• .ci/jenkins/pipeline/Jenkinsfile
• .ci/jenkins/pipeline/Jenkinsfile.dispatcher
• .ci/jenkins/pipeline/proj-jjb.yaml

💤 Files with no reviewable changes (1)

• .ci/jenkins/pipeline/Jenkinsfile

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.ci/jenkins/pipeline/Jenkinsfile.dispatcher:
- Around line 27-33: Guard the GitHubHelper initialization in the dispatcher
flow so manual runs do not fail when VARIABLE_FROM_POST is empty. In the
Jenkinsfile.dispatcher setup around GithubHelper.getInstance and
currentPrNumber, skip the GitHub-specific credential/helper path when there is
no payload, or ensure manual dispatches always provide a payload before calling
getPRNumber.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 5a235188-9c3d-4af4-b9e6-20b0a725b14b

📥 Commits

Reviewing files that changed from the base of the PR and between e0a17e5 and 996aa8d.

📒 Files selected for processing (4)

• .ci/docs/ci-overview.md
• .ci/jenkins/pipeline/Jenkinsfile
• .ci/jenkins/pipeline/Jenkinsfile.dispatcher
• .ci/jenkins/pipeline/proj-jjb.yaml

💤 Files with no reviewable changes (1)

• .ci/jenkins/pipeline/Jenkinsfile

[svc-nixl]

👀 Investigating nixl-ci-gpu build 2553…

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 038bd648

Now I have enough evidence. Let me analyze the timeline carefully.

Timeline analysis:

• The test suite test/python/test_nixl_bindings.py completes successfully around 13:53:37.
• Then setup of test_empty_notif_tag (in test_nixl_api.py) begins. Its fixture two_connected_agents creates a nixl_agent, which calls create_backend("UCX") → agent.createBackend("UCX", ...).
• The failure is: Failed: Timeout (>5.0s) from pytest-timeout at _api.py:383 inside create_backend.
• Final line: 23 passed, 2 skipped, 1 error in 239.52s.

This is not a wall-clock kill of the whole job — the individual test's createBackend call hung and was killed by the pytest-timeout 5-second per-test limit. The UCX backend instantiation (agent.createBackend) blocked for >5s. Throughout the run, hundreds of ibv_query_port_speed(...) failed: Protocol not supported warnings are emitted (a UCX/IB environment quirk), but the actual hang is inside UCX createBackend. Since it's the setup fixture that timed out (not a slow-but-progressing test), this is a hang in UCX backend creation, not a case needing more time.

Note: The jnlp container args contain a JENKINS_SECRET value marked ******** (already redacted by Jenkins) — no action needed, but that env block should not be echoed.

Summary: The Python binding test test_nixl_api.py::test_empty_notif_tag errored during fixture setup because nixl_agent.create_backend("UCX") hung and was killed by the 5s pytest-timeout.

Root cause: agent.createBackend("UCX", {}) (called from _api.py:383) blocked for more than the 5-second per-test timeout during instantiation of a second UCX-backed agent in the two_connected_agents fixture. This is a hang inside UCX backend creation, not a genuinely slow test — the last application log before the pytest-timeout is the createBackend call, and every prior test's UCX init produced only the benign ibv_query_port_speed(...) failed: Protocol not supported warnings, meaning the IB port-speed query path in UCX is degraded/slow on this node (mizu01, mlx5_0..3 IB ports report "Protocol not supported"). Under that condition, UCX device/interface discovery during createBackend intermittently stalls past 5s. The earlier test_nixl_bindings.py agents happened to finish just under the limit; the fixture-created agent in test_nixl_api.py did not.

Implicated commit: unknown — the failure is in the UCX backend createBackend path (surfaced through src/api/python/_api.py:383 → nixl_agent.cpp:createBackend); no single recent commit in the retrievable history clearly introduces it, and this build is PR #1889 (commit 038bd64). Cannot attribute with confidence from logs alone.

File: src/api/python/_api.py:383 (self.backends[backend] = self.agent.createBackend(backend, initParams)); test entry point test/python/test_nixl_api.py:58 (two_connected_agents fixture).

Suggested fix:

• Investigate why UCX createBackend stalls on mizu01: the repeated ibv_query_port_speed(mlx5_*:1/IB) failed: Protocol not supported indicates the IB ports are in an unsupported/misconfigured state for this UCX build (v1.21.x). Confirm the node's RDMA/IB fabric and driver state, or constrain UCX to usable transports for CI (e.g., set UCX_TLS to exclude the failing IB devices, or UCX_NET_DEVICES to a working device) so device discovery in createBackend doesn't stall.
• Do not simply raise the pytest-timeout — the operation is hanging, not legitimately slow. If a guard is desired, add a bounded timeout/retry inside backend init and surface a clear error instead of blocking.
• Reproduce locally with the same UCX v1.21.x build and UCX_LOG_LEVEL=debug to capture where createBackend blocks (device probe vs. worker/endpoint setup).

Related: none found via issue/PR search (ovidiusm PR #1546 "Defer plugin load to the point where they are actually used" touches the plugin/backend init path in _api.py and is worth reviewing as a possible interaction point, but not confirmed as the cause).