device api: Restore user-facing Device API Doxygen comments

[tomerdav]

Summary

• Restore the original user-facing GPU Device API Doxygen comments that were dropped when device api: Layer UCX device API from GPU API #2147 moved the public API out of the UCX header.
• Document both the namespaced nixl::gpu API and the C-style wrappers (nixlPut, nixlGpuGetXferStatus, nixlAtomicAdd, nixlGetPtr).
• Include the public device headers in Doxygen so host APIs that @ref nixlPut / @ref nixlAtomicAdd can resolve.

Test plan

• Review src/api/device/gpu/nixl_device.cuh and device_types.cuh against the pre-device api: Layer UCX device API from GPU API #2147 comments on src/api/gpu/ucx/nixl_device.cuh
• Optional: generate Doxygen and confirm nixlPut / nixl::gpu::put appear in the output
• Confirm no functional/code changes beyond comments and Doxygen input wiring

Summary by CodeRabbit

• Documentation
  • Improved API documentation for GPU device types, transfer statuses, memory views, and related public interfaces.
  • Expanded documentation for GPU memory transfers, atomic operations, remote-pointer access, and C-style wrappers.
  • Updated documentation generation to recognize CUDA header files and include relevant GPU headers.
  • Generated reference materials now provide broader coverage of GPU-related APIs and data types without changing runtime behavior or public API signatures.

[github-actions]

👋 Hi tomerdav! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 9f3296cc-91cb-45f8-a93d-e03cf893c06d

📥 Commits

Reviewing files that changed from the base of the PR and between 5bf9507 and 813da51.

📒 Files selected for processing (1)

• src/api/device/gpu/device_types.cuh

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Walkthrough

Walkthrough

Doxygen now parses CUDA headers as C++ and scans the GPU API directory. The GPU type and device headers now document public transfer types, memory views, status queries, transfers, atomic operations, pointer lookup, and legacy wrappers.

Changes

GPU API documentation

Layer / File(s)	Summary
Doxygen CUDA header configuration Doxyfile	Doxygen parses .cuh files as C++ and scans the GPU API directory.
GPU type reference documentation src/api/device/gpu/device_types.cuh	Added documentation for transfer status, transfer levels, defer flags, memory views, and public aliases.
GPU device API reference documentation src/api/device/gpu/nixl_device.cuh	Added documentation for status retrieval, GPU transfers, atomic operations, remote-pointer lookup, and legacy wrappers. Runtime behavior and public API signatures remain unchanged.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: vvenkates27

Merge Risk: ⚪ Minimal · up to 92197

This change restores GPU API reference documentation and Doxygen visibility without changing runtime behavior, so it is ready to merge.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains the restored Doxygen documentation, the reason for the change, and the validation plan. It does not use the repository template headings, but it contains the required informat…
Title check	✅ Passed	The title clearly identifies the main change: restoring user-facing Device API Doxygen comments. It is concise and specific.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 5bf95076

TL;DR: The build never got past Checkout source code — every git fetch against https://github.com/ai-dynamo/nixl.git returned HTTP 401 with No credentials specified, so all 6 checkout retries failed and Jenkins aborted. This is a Jenkins SCM credential problem, not a defect in the commit under test.

Full analysis

Summary: nixl-ci-non-gpu #2963 failed in its only stage, "Checkout source code" (55s), with hudson.plugins.git.GitException: ... returned status code 128: error: RPC failed; HTTP 401 curl 22 ... fatal: expected flush after ref listing, repeated 6× until "Maximum checkout retry attempts reached, aborting".

Root cause: The git checkout ran anonymously — the log prints No credentials specified before each attempt — and GitHub rejected the unauthenticated ref advertisement with HTTP 401. The pipeline's SCM configuration for this job supplied no credentialsId (or the referenced credential/GitHub App token has been rotated, revoked, or expired, leaving the binding empty), so git fetch --tags --force --progress -- https://github.com/ai-dynamo/nixl.git +refs/heads/*:refs/remotes/origin/* had nothing to authenticate with. Evidence that this is not a transient network blip: all six attempts, 10s apart, failed with the identical 401 at the ref-listing phase — a rate-limit or outage would normally vary (403/5xx/timeouts), and an unauthenticated 401 is a deterministic auth rejection. Nothing in the build reached compile or test, and the commit under test (5bf9507, branch tdavidor/device-api-restore-docs, a docs change) was never fetched, so it cannot be implicated.

Implicated commit: unknown — no repository commit is implicated; the failure precedes checkout of [REDACTED:Hex High Entropy String]. The cause lies in Jenkins job/credential configuration outside the repo.

File: Jenkins job configuration for nixl-ci-non-gpu (SCM / checkout scm credentials binding); no repo file at fault. Worker: worker-[REDACTED:Hex High Entropy String]-26ed-4853-a7b0-b033c9503430-4s1mc-g5sx1.

Suggested fix: Have a Jenkins admin re-attach a valid credential to the nixl-ci-non-gpu SCM definition and verify the underlying secret is still live — check the GitHub App installation token / PAT used by the nbu-swx-nixl-main folder for expiry or revocation and rotate it if expired (do not paste the value anywhere). Concretely: (1) confirm the job's checkout/multibranch source specifies a non-empty credentialsId rather than falling through to anonymous, since No credentials specified indicates it currently does not; (2) after rotating, re-run #2963 to confirm. Also worth checking whether other jobs in the same folder polling GitHub are failing identically — if so, the shared folder-level credential is the single point of failure and should be fixed there rather than per-job. Longer term, make the checkout stage fail fast with a clear message on 401 instead of burning 6 retries, so credential expiry is distinguishable from network flakiness at a glance.

Related: none — searched issues/PRs for prior checkout-401/credential failures and found no matching reports; recent .ci history (most recently 8585eee0, Anant Sharma, "ci: exclude organization members from external label") contains no change to checkout credentials.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 8c022be6-cc23-4d9d-90c6-3742be6d800e in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-test-sanitizers · commit 5bf95076

TL;DR: The build never got past SCM checkout — every git fetch against https://github.com/ai-dynamo/nixl.git was rejected with HTTP 401, because the checkout ran with "No credentials specified" and the agent's HTTPS credential is missing/expired. This is a Jenkins credential/infra problem, not a defect in commit 5bf9507 or PR #2230.

Full analysis

Summary: nixl-ci-test-sanitizers #1059 failed in the only stage that ran, "Checkout source code" (58 s), after 5 retry attempts of git fetch all failed with RPC failed; HTTP 401 curl 22 ... fatal: expected flush after ref listing, ending in ERROR: Maximum checkout retry attempts reached, aborting.

Root cause: The pipeline's checkout step is configured without SCM credentials — the log states No credentials specified before each attempt — so the agent pod (worker-4d748035-e011-4f8a-899f-[REDACTED:Hex High Entropy String]-l2cjj-rrv8n) relies on ambient auth (a credential helper / GitHub App or PAT injected into the agent image, or an authenticating egress proxy). That auth was rejected: GitHub answered 401 during the ref-listing phase of git fetch --tags --force --progress -- https://github.com/ai-dynamo/nixl.git +refs/heads/*:refs/remotes/origin/*. The failure is intermittent rather than absolute — attempt #4 completed the initial ref fetch and got as far as git config remote.origin.url, then failed with the same 401 when fetching the PR merge ref +[REDACTED:Hex High Entropy String]:refs/remotes/origin/.... That pattern (some requests pass, most 401) is characteristic of an expired/rotated GitHub App installation token or a revoked PAT still cached in the agent, or a proxy dropping the auth header — not of a bad ref or bad commit. No source was ever fetched, no build or sanitizer test executed, and no artifacts were produced, so nothing in the tree under test can have contributed.

Implicated commit: unknown — no repository content was fetched; the triggering commit 5bf9507 / merge ref cac415a is not implicated.

File: unknown — failure is in the Jenkins job's SCM configuration (the checkout/GitSCM step of the nixl-ci-test-sanitizers job definition), not in a repository file.

Suggested fix: Have CI infra rotate and re-attach the GitHub HTTPS credential used by the sanitizer agents, and make the checkout use it explicitly instead of ambient auth:

1. In the nixl-ci-test-sanitizers job, set an explicit credentialsId on the userRemoteConfigs entry for https://github.com/ai-dynamo/nixl.git so the log no longer reports No credentials specified; if a GitHub App credential is used, verify the installation still grants access and that its token is being minted per-build rather than cached in the agent image.
2. Check the agent pod's egress path — if an authenticating HTTPS proxy fronts github.com, confirm http_proxy/https_proxy and any ~/.git-credentials or credential.helper in the agent image are valid; a stale helper entry produces exactly this 401 at ref listing.
3. Treat any credential seen in the agent configuration as compromised if it is a long-lived PAT and rotate it.
4. Once the credential is fixed, simply re-run build DOCA 3.2 GPUNetIO on CI #1059 — no code change is needed. Separately, the job also emits DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs; switching to deleteDir() would remove the repeated full re-clone and reduce exposure to transient auth failures.

Related: none — searching the project for this signature returned only an unrelated match (PR #401), and no existing issue tracks the 401 checkout failure.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 846495c8-3be8-4732-a8e8-63b1de7bd294 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 5bf95076

TL;DR: The build never got past Checkout source code — every git fetch from https://github.com/ai-dynamo/nixl.git was rejected with HTTP 401, so Jenkins exhausted its 5 checkout retries. This is a CI credential/proxy problem on the agent, not a defect in PR #2230.

Full analysis

Summary: Stage 13 "Checkout source code" failed after 60s; all clone/fetch attempts died with error: RPC failed; HTTP 401 curl 22 The requested URL returned error: 401 → ERROR: Maximum checkout retry attempts reached, aborting.

Root cause: The Git SCM step runs with No credentials specified (visible before each attempt) and the HTTPS fetch to github.com returns 401 Unauthorized on every one of the 5 retries, including a plain git fetch --tags --force --progress -- https://github.com/ai-dynamo/nixl.git +refs/heads/*:refs/remotes/origin/* and the pinned-SHA fetch +[REDACTED:Hex High Entropy String]:refs/remotes/origin/.... A 401 (not 404) on an anonymous fetch of a public repo means an authenticating intermediary or a stale credential is in play on the agent pod worker-e84ae507-a49e-459b-9f49-[REDACTED:Hex High Entropy String]-6mrbv-g7rnt — i.e. an expired/revoked token being injected via http.extraheader, a credential.helper / ~/.gitconfig url.insteadOf rewrite, or an authenticated HTTPS proxy (http_proxy/https_proxy) that the pod image now points at. The failure is deterministic across all retries and identical for both the branch-glob and SHA-pinned fetch, so it is not transient network flake and not rate limiting.

Two corroborating details:

• The failure is entirely inside SCM checkout — no repository content was ever fetched, so no source file from commit 5bf9507 was compiled or tested. The branch (tdavidor/device-api-restore-docs, a docs-only PR) cannot have caused it.
• The pipeline also asked for SHA [REDACTED:Hex High Entropy String], which differs from the trigger commit 5bf9507; worth confirming the poll/trigger wiring, but the 401 blocks even the unpinned fetch, so it is secondary.

Implicated commit: unknown — no repo commit is implicated; the failure precedes checkout. Suspect a change to the Jenkins agent image / job credential binding rather than a source commit.

File: N/A (Jenkins job SCM configuration for nixl-ci-dl-gpu; failure at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1220))

Suggested fix:

1. Attach a valid credential to the Git SCM step of nixl-ci-dl-gpu (the log's No credentials specified should not be there for an HTTPS clone through an authenticating path), and rotate the GitHub token/PAT used by this job — a 401 is the signature of a revoked or expired token. Do not reuse the old one.
2. On the agent image, inspect and clean any injected auth: git config --global --list for http.extraheader, credential.helper, and url.<...>.insteadOf rewrites, plus http_proxy/https_proxy/no_proxy env. A leftover http.extraheader with a dead token reproduces exactly this error.
3. As a quick verification, run git ls-remote https://github.com/ai-dynamo/nixl.git from a shell in that agent pod — if it also 401s, the problem is agent-level config/proxy, confirming it is independent of the job.
4. Once fixed, simply re-run build Marvell ODM dma-buf NIXL plugin #2183; no code change to PR device api: Restore user-facing Device API Doxygen comments #2230 is needed.

Related: none found (searched the repo for this error signature; only unrelated PRs #401 and #2085 matched).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 94f7efe6-2ecb-4221-a02b-097935d37fd0 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 5bf95076

TL;DR: The build never got past "Checkout source code" — every git fetch from https://github.com/ai-dynamo/nixl.git on the agent pod returned HTTP 401 (5 retries, then abort). This is a Jenkins agent credential/proxy problem, not a code defect in PR #2230.

Full analysis

Summary: Stage 13 "Checkout source code" failed after 5 retries with hudson.plugins.git.GitException: ... returned status code 128: error: RPC failed; HTTP 401 curl 22 ... fatal: expected flush after ref listing.

Root cause: The git SCM step ran with No credentials specified (anonymous clone of a public repo), yet the remote answered 401 Unauthorized on the ref-listing request. Anonymous access to a public GitHub repo does not return 401 — a 401 means invalid/expired credentials were sent on the wire. So something in the agent environment supplied bad auth: a stale token in a global git config on the agent image (http.extraheader, credential.helper, ~/.git-credentials), a GIT_ASKPASS/GITHUB_TOKEN env var baked into the pod template, or an authenticating HTTPS proxy in front of github.com. All 5 attempts failed identically on the same pod (worker-0dc80ebb-40de-4afd-9b4c-[REDACTED:Hex High Entropy String]-x6dqg-j51fk), and the two clone flavours (+refs/heads/* and the extra [REDACTED:Hex High Entropy String]:refs/remotes/origin/... merge-ref fetch) failed the same way — the failure is independent of the ref being fetched, i.e. independent of the PR content.

Corroboration that this is not the PR: build #556 of the same job completed "Checkout source code" successfully in 10.7 s and proceeded into the parallel "Setup docker"/"Build image" stages. Nothing in the log touches nixl source at all — no compiler, no test, no repo file was ever read.

Implicated commit: none — the failure precedes checkout of any source. Commit [REDACTED:Hex High Entropy String] (branch tdavidor/device-api-restore-docs, PR #2230) is not implicated.

File: not a source file — Jenkins job SCM configuration / agent pod template for nixl-ci-build-container-pr (failing agent: worker-0dc80ebb-40de-4afd-9b4c-[REDACTED:Hex High Entropy String]-x6dqg-j51fk)

Suggested fix:

1. Simply re-run build NIXLBench: Align memory for O_DIRECT #557 first — if it lands on a healthy agent it will pass, which confirms a single bad/misconfigured pod.
2. On the agent image/pod template, inspect and clean up injected git auth: git config --global --list | grep -Ei 'extraheader|credential|proxy', plus ~/.git-credentials, GIT_ASKPASS, GIT_CONFIG_*, http_proxy/https_proxy. Remove any expired token. If a token was baked into the image, rotate it — do not just delete the reference.
3. Preferably attach an explicit, valid credential to the SCM step instead of relying on anonymous access, so a broken ambient credential can't shadow it (the log's No credentials specified is the tell).
4. Housekeeping surfaced by the same log: the job still uses the deprecated "Wipe out repository & force clone" extension (DEPRECATED: ... Pipeline users should use the deleteDir() step instead) — worth migrating, and it also makes every retry pay a full clone.

Related: none — issue/PR search for this signature returned only unrelated matches (#401, #2085, #2123).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 1c5f2ca3-2620-41dc-9f32-c15bef80ca79 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit 5bf95076

TL;DR: The build never got past Checkout source code — all five Git SCM clone attempts of https://github.com/ai-dynamo/nixl.git failed with HTTP 401 ("No credentials specified"), so this is a Jenkins credential/network-auth infra failure, not a defect in PR #2230.

Full analysis

Summary: nixl-ci-dl-gpu-ep #1088 failed in its only stage, Checkout source code (55 s), with hudson.plugins.git.GitException ... git fetch ... returned status code 128: error: RPC failed; HTTP 401 curl 22 ... fatal: expected flush after ref listing, repeated 5× then ERROR: Maximum checkout retry attempts reached, aborting.

Root cause: The Git SCM step on agent pod worker-73e99e7b-9636-4a7b-a9b7-[REDACTED:Hex High Entropy String]-0fdkc-fm1cn ran with No credentials specified (no credentialsId bound to the SCM config) and the HTTPS fetch to github.com was rejected with 401 Unauthorized during ref-listing. Since ai-dynamo/nixl is reachable anonymously, a 401 means invalid/expired credentials were injected into the request — i.e. a stale GitHub App/PAT token or an http.extraHeader/credential-helper (or authenticating proxy) on the agent image supplying bad auth — rather than the request being anonymous. The failure is fully contained in checkout: no build, test, or repo code was ever executed, so commit 5bf9507 (tdavidor/device-api-restore-docs, a docs-only change) cannot be implicated. The consistent, non-flaky 401 across all 5 retries (10 s apart) points at a persistent credential state, not a transient network blip.

Implicated commit: unknown — no repo commit is implicated; the failure precedes checkout of the tree. (For context only, the most recent CI-credential-related change in history is d6187c6e "CI: Suppress credentials in logs (#2074)" by Noam Tsemah, and cabe5c24 by Alexey Rivkin touched .ci/jenkins most recently; neither is confirmed as the cause from this log.)

File: Jenkins job configuration for nixl-ci-dl-gpu-ep (SCM/checkout section) — not a file in the repo; the referenced pipeline definition under .ci/jenkins/pipelines/ for this job was not readable at the expected path.

Suggested fix: This needs a CI-admin action, not a code change:

1. Re-attach a valid credentialsId to the job's Git SCM config (or to the checkout/git step in the job's pipeline definition) instead of relying on anonymous access — the log explicitly reports No credentials specified.
2. Rotate/renew the GitHub App installation token or PAT used by this Jenkins instance and verify it is not expired or revoked; do not paste the token value anywhere in build logs or comments.
3. On the agent image, check for a leftover http.extraHeader / credential.helper in /etc/gitconfig or ~/.gitconfig that would inject stale auth into the fetch, and for an authenticating egress proxy (http_proxy/https_proxy) in the pod env.
4. Also replace the deprecated "Wipe out repository & force clone" extension with deleteDir() as the log warns, while touching this config.
5. Once credentials are fixed, simply re-trigger Cyclic buffer telemetry exporter moved to use plug-in infrastructure #1088 for commit 5bf9507; no change to PR device api: Restore user-facing Device API Doxygen comments #2230 is required.

Related: PR #2230 (trigger, docs-only — not the cause). No existing issue matches this 401 checkout signature; a search for it returned only an unrelated PR (#401).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 77a795ca-e213-4b09-8a94-f945f54a740f in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit 5bf95076

TL;DR: The build failed in the build_helper_vllm "Prepare" stage because the container-registry login to artifactory.nvidia.com died with net/http: TLS handshake timeout (podman exit 125) — a transient registry/network flake unrelated to the PR; wrap that login in a retry loop and re-run.

Full analysis

Summary: Stage 414 ("Prepare", the build_helper_vllm parallel branch) failed with exit code 125 while authenticating to artifactory.nvidia.com, aborting the whole nixl-ci-build-wheel #1676 pipeline.

Root cause: Infrastructure/network flake, not a code defect. The Prepare step's registry login produced:

• Error: authenticating creds for "artifactory.nvidia.com": Get "https://artifactory.nvidia.com/v2/token?account=svc-nixl&service=artifactory.nvidia.com": net/http: TLS handshake timeout

The TLS handshake to Artifactory did not complete within the Go client's default 10s window (login started ~11:57:27, failed 11:57:39). Evidence that this is transient rather than a config/credential problem:

• The sibling Prepare stage (node 404) ran the exact same login one second earlier on the same pod spec and printed Login Succeeded!.
• Both stages emitted the same benign unlinkat /var/lib/containers/storage: device or resource busy / umount ... no such file or directory podman shutdown noise, including the one that succeeded — so that is not the cause.
• Everything else in the pipeline worked, including two full Build Wheel stages, Build sanity image (which pulled and pushed to the same Artifactory host, using a for i in {1..6} retry loop that succeeded on the first attempt), and Run vLLM sanity which ended with [sanity:vllm] PASS. There is no timestamp gap or hang anywhere — the build was continuously active; the only failure is the 12-second login timeout.
• The PR content (tdavidor/device-api-restore-docs, device api: Restore user-facing Device API Doxygen comments #2230) is documentation-only and cannot affect registry authentication.

Implicated commit: none — [REDACTED:Hex High Entropy String] is not implicated; this is an infrastructure failure.

File: the Jenkins pipeline "Prepare" step that runs podman/docker login artifactory.nvidia.com (under .ci/jenkins/; the same helper used by the successful node-404 Prepare). Recent CI-config touches for context: cabe5c24 (Alexey Rivkin), d309bc29 (NirWolfer), c984ce65 (lishapira).

Suggested fix:

1. Retry the build now — this will almost certainly pass, as the parallel branch's identical login succeeded.
2. Durable fix: make the Prepare login resilient the same way the image push already is. Wrap it in a bounded retry with backoff, e.g.:

   for i in $(seq 1 6); do
     podman login --username "$ARTIFACTORY_USER" --password-stdin artifactory.nvidia.com <<<"$ARTIFACTORY_TOKEN" && break
     [ "$i" = 6 ] && exit 1
     sleep $((i * 5))
   done

   The push step in Build sanity image already uses for i in {1..6} + break; the Prepare login has no such protection, which is why a single 10s TLS timeout kills a ~30-minute pipeline.
3. Optionally raise the client TLS/HTTP timeout for the login call, and consider failing that parallel branch soft (catchError) so a registry hiccup in one matrix leg doesn't abort legs that already passed.

Note: the Artifactory credential is correctly masked in this log (Masking supported pattern matches of $ARTIFACTORY_TOKEN); no secret exposure to remediate.

Related: none — no existing issue/PR tracks this login flake. (Search hits #2007 and #2147 are unrelated.)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id c2c29bb3-44bb-452c-9398-b05ef56bd02c in the triage console for the audit trail.

[tomerdav]

/build

[ColinNV]

I believe device_types.cuh should have an @file.

[tomerdav]

I believe device_types.cuh should have an @file.

Added a file-level Doxygen description to device_types.cuh.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 813da515

TL;DR: One of the six matrix axes failed its Build step after 4 s with .gitlab/build.sh: No such file or directory (exit 127) — that pod's workspace was empty (stashed source never landed there), not a code defect; retrigger the build and add a workspace-populated guard/retry around the per-pod unstash.

Full analysis

Summary: nixl-ci-non-gpu #2995 aborted because one parallel matrix branch's Build stage (node id 598) exited 127: /home/jenkins/agent/workspace/nixl-ci-non-gpu@tmp/durable-b5daa5eb/script.sh.copy: line 4: .gitlab/build.sh: No such file or directory.

Root cause: Infrastructure/workspace race, not a source problem. The pipeline checks out once ("Checkout source code", stage 13): it wipes /home/jenkins/agent/workspace/nixl-ci-non-gpu/*, clones [REDACTED:Hex High Entropy String], runs .ci/cidemo-init.sh, then tar -c ... -f scm-repo.tar . and stashes it; each matrix pod re-materialises the tree in its own workspace. In the failing pod that tree was absent when the step ran — the stage produced exactly one line and lasted 4.2 s, with none of the normal preamble (nixl_concurrent_id=0, + set -o pipefail) that the other axes show. The same script from the same commit built fine in the other five axes (e.g. stage 514, aarch64, 163 s, full meson/ninja build + install), so .gitlab/build.sh clearly exists in the checkout. Build #2994 passed all six axes with the identical matrix, confirming a one-off. The failing branch's exit 127 then propagated: hudson.AbortException: parallel task failed ... Step Build failed with exit code=127. PR #2230 is documentation/Doxygen-comment only and cannot produce a missing-file error at shell level.

Implicated commit: none — [REDACTED:Hex High Entropy String] (PR #2230, tomerdav) is not implicated; the failure is in the CI workspace-provisioning path.

File: .ci/jenkins/lib/build-matrix.yaml:78 (.gitlab/build.sh ${NIXL_INSTALL_DIR} — the step that failed); source of the tree: checkout/stash of scm-repo.tar in the "Checkout source code" stage.

Suggested fix:

1. Retrigger #2995 — the failure is not reproducible from the commit under test.
2. Harden the per-pod source provisioning in the shared pipeline: make the unstash/untar of scm-repo.tar happen inside the retry { node { ... } } scope so a replaced/reconnected agent pod re-populates its workspace, and add a preflight assertion before the first step, e.g. test -f .gitlab/build.sh || { echo "FATAL: workspace not populated (scm-repo.tar unstash missing)"; exit 1; }, so the symptom is reported as a provisioning error instead of a bare exit 127.
3. Optional secondary cleanup seen in the passing axes (unrelated to this failure, but from PR device api: Restore user-facing Device API Doxygen comments #2230's Doxygen wiring): doxygen prints warning: tag INPUT: input source 'docs/doxygen/nixl_doxygen.md' does not exist, same for README.md, LICENSE, ATTRIBUTIONS.md, CONTRIBUTING.md, plus error: Extra file 'docs/doxygen/nixl.png' ... does not exist! — these paths are being resolved relative to nixl_build/ rather than the source root and should be made absolute (@SOURCE_ROOT@/...) in the generated Doxyfile.configured.

Related: PR #2230 (build under test); #2147 (the refactor whose comments PR #2230 restores). No existing issue found for the .gitlab/build.sh: No such file or directory CI flake — worth opening one against the CI pipeline if it recurs.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id abf44da8-07a2-4391-b30a-72fe8057c323 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit 813da515

TL;DR: Build #3532's "Run CPP tests" stage was SIGTERM'd (exit 143) at test 246/255 because the UCX/etcd-dependent gtests ran 5–20× slower than baseline, not because of a hang or the PR's changes — three nixl-ci-gpu builds (3531, 3532, 3535) were allocated the same slurm node mizu01 at overlapping times. Serialize/exclusive-allocate the GPU CI node (or split the gtest stage's time budget) and re-run; PR #2230 is docs-only.

Full analysis

Summary: Stage Run CPP tests (node 202) was aborted after 61 min ("Sending interrupt signal to process" → script returned exit code 143) while gtest-parallel was at [246/255] ucx_tracing_nsys_auto/..., so the remaining device-API tests never ran.

Root cause: Wall-clock kill, but not a hang — the log shows continuous per-test progress right up to 14:56:10, and the largest inter-line gap (~2.4 min, 14:41:30→14:43:51) is exactly one test's own self-reported 144806 ms runtime. The stage overran because the suite was pathologically slow. Comparing against build #3531, which ran the identical 255-test suite on the same host earlier the same hour:

test	#3531	#3532
ucx/TestErrorHandling.BasicXfer/0	3583 ms	51816 ms
ucx_threadpool/TestErrorHandling.XferFailRestore/0	11558 ms	149669 ms
MDManagerFixture.SendLocalWithoutWorker	3574 ms	38217 ms
MetadataExchangeTestFixture.EtcdSendLocalAndFetchRemote	42956 ms	64858 ms
MpStoreTest.*, Config.*, HardwareWarningTest.* (pure CPU)	441–1587 ms	441–1607 ms

Pure-CPU tests are bit-for-bit the same duration; only agent/connection-establishing tests (UCX, etcd, metadata exchange) blow up by a fixed additive penalty. That signature is environmental contention on the shared network/IB/GPU fabric, not a code regression — and the commit under test is a docs restore on tdavidor/device-api-restore-docs.

Infra telemetry corroborates the contention: slurm_metrics.slurm_jobs_all_local shows allocation 91984 nixl-ci-gpu-3532 on mizu01 (13:56:07 → CANCELLED 14:57:52, 24 CPUs / 4 GPUs) overlapping 91976 nixl-ci-gpu-3531 on mizu01 (13:46:33 → CANCELLED 14:24:09, 24 CPUs / 4 GPUs) and then 92021 nixl-ci-gpu-3535 on mizu01 (from 14:45:17). Three CI builds, all user svc-nixl, sharing one host that has only 8 GPUs; pyxis/enroot containers share the host network namespace, so their UCX endpoints, IB devices and etcd instances collide. slurm_nodes_local for mizu01 over 13:50–15:10 shows cpu_load ≤ 3.7 of 256 CPUs and ~840 GB memory free, so this is not CPU or RAM starvation — it is fabric/device-level contention between the co-scheduled jobs. Notably the tail of the run (14:46 onward, after 3531 fully released) returns to near-baseline timings (ErrorCallbackMarksEndpointFailedWithoutClosingIt 3748 ms vs 2422 ms baseline), which fits the contention explanation.

Implicated commit: none — no code commit implicated; 813da51 (PR #2230, docs-only) is not the cause.

File: CI job configuration for nixl-ci-gpu (the srun ... --oversubscribe allocation used in the Allocate Environment / Run CPP tests stages); not a repo source defect.

Suggested fix:

1. Stop co-scheduling nixl-ci-gpu builds on one node: request exclusive allocation (--exclusive, drop --oversubscribe) or add a Jenkins lock/throttle so only one nixl-ci-gpu slurm job per node is in flight. Allocation 91984 on mizu01 is what an admin needs to trace this.
2. Re-run build #3532; the suite passes 246/255 in ~25 min when the node is not shared.
3. Separately, give the gtest step its own timeout and fail with a clear message rather than a bare 143, and consider trimming the 4× ucx / ucx_no_pt / ucx_threadpool / ucx_threadpool_no_pt TestErrorHandling matrix, which alone is ~35 min of the stage.
4. Follow-up (independent bug, visible in #3531): ucxDeviceApi/SingleWriteTest.SingleWorkerPutGap/{UCX_BLOCK,UCX_WARP,UCX_THREAD} fails at test/gtest/device_api/single_write_test.cu:538 (ASSERT_NE(ptr, nullptr) — "nixlGetPtr is null unless the remote is locally mapped (cuda_ipc / NVLink)"), which is expected to recur once #3532 reaches tests 253–255; this is plausibly also a symptom of only 4 of 8 GPUs being allocated to the job.

Related: PR #2230 (build trigger, docs-only); prior build #3531 on the same node for the timing baseline and the device-API test failure. Possibly relevant existing work: #1743 "TEST/GTEST: Run in single process."

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 59364435

TL;DR: All four Build image variants fail at Dockerfile step 53/72 because the torch install derives its wheel index from CUDA_VERSION (13.4 → https://download.pytorch.org/whl/cu134), an index that has no cp312 torch wheels, so uv pip install torch errors out; pin the index to a real one (e.g. cu130/cu129) instead of computing it.

Full analysis

Summary: nixl-ci-build-container-pr #619 — every parallel Build image stage (x86_64 and aarch64) fails building contrib/Dockerfile at STEP 53/72 (torch install), exit status 1.

Root cause: The Dockerfile computes the PyTorch wheel index from the base image's CUDA version: export UV_INDEX="https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d .)". With BASE_IMAGE_TAG=26.08-cuda13.4-devel-ubuntu24.04, CUDA_VERSION=13.4 yields cu134, for which PyTorch publishes no matching wheels. uv reports exactly this:

× No solution found when resolving dependencies:
╰─▶ Because all versions of torch have no wheels with a matching Python ABI tag (e.g., `cp312`) …
hint: `torch` was found on https://download.pytorch.org/whl/cu134, but not at the requested version …
hint: You require CPython 3.12 (`cp312`), but we only found wheels for `torch` (v2.0.1) with … `cp38`, `cp39`, `cp310`, `cp311`

This is environment-independent (it reproduces identically on x86_64 and aarch64 stages), not a node/infra issue. The failure was introduced when the base images were bumped to CUDA 13.4 while the index remained derived from CUDA_VERSION.

Implicated commit: [REDACTED:Hex High Entropy String] — "build: bump CUDA and CI base images, stop restating them across CI (#2205)", NirWolfer (bumped BASE_IMAGE_TAG to 26.08-cuda13.4-*; the fragile index derivation is on the line below).

File: contrib/Dockerfile:333-338 (specifically line 336, the UV_INDEX construction; base tag at contrib/Dockerfile:17)

Suggested fix: Stop deriving the index from CUDA_VERSION. Either:

1. Add an explicit build arg, e.g. ARG TORCH_INDEX_URL="https://download.pytorch.org/whl/cu130", and use export UV_INDEX="${TORCH_INDEX_URL}"; or
2. Map the CUDA major/minor to the newest index PyTorch actually publishes (fall back to the CUDA-major index, cu13/cu130) and add --index-strategy unsafe-best-match plus a hard failure if no wheel resolves, so the mismatch is obvious rather than silently resolving to torch 2.0.1.

Note PR #2249 ("build: pin the torch wheel index instead of deriving it from CUDA_VERSION") already implements this; land it (or rebase this PR on it) to unblock the container build.

Related: PR #2249 (fix in flight); PR #2205 (base image bump that exposed it); this build's PR #2230.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id a2709aeb-953c-4b78-afff-902f99d8378d in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 1d2473cb

TL;DR: Two of the six parallel Build variants failed in meson setup because the taskflow subproject is fetched with a [wrap-git] clone from github.com, and in those two containers the clone was rejected ("could not read Username for 'https://github.com'"); convert subprojects/taskflow.wrap to a hashed [wrap-file] tarball (like liburing/tomlplusplus) or pre-provision taskflow in the CI images.

Full analysis

Summary: nixl-ci-non-gpu #3047 failed in the parallel matrix at stages Build (node 358) and Build (node 394) — meson setup nixl_build … aborted while resolving the taskflow fallback subproject.

Root cause: meson.build:233 requires dependency('taskflow', fallback: ['taskflow','taskflow_dep']), and subprojects/taskflow.wrap is a [wrap-git] entry pointing at https://github.com/taskflow/taskflow.git. In the two failing containers the clone failed immediately:

Cloning into 'taskflow'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
...
ERROR: Subproject taskflow is buildable: NO
meson.build:233:16: ERROR: Git command failed: ['/usr/bin/git', '-c', 'advice.detachedHead=false', 'clone', '--depth', '1', '--branch', 'v3.10.0', 'https://github.com/taskflow/taskflow.git', 'taskflow']

Git was pushed into interactive credential prompting (i.e. the anonymous HTTPS fetch was refused/redirected) and, with no tty, died instantly. Note that in the same build the tarball-based wraps worked fine in these containers (Downloading liburing source from https://github.com/axboe/liburing/... succeeded), and in the passing variant (stage 403) the same git clone succeeded and took ~27 s (Cloning into 'taskflow'... 11:32:53 → subproject executed 11:33:19). So the failure is in the git-transport dependency fetch, not in the code under test — this is a docs-only PR (#2230, Doxygen comments) and every Test stage that got past configure passed. No hang: the failing stages died 11 s and 4 s after start with continuous output.

Implicated commit: 76275cf — "Use tagged Taskflow git wrap for NIXL builds (#2121)", author bzsuni (converted taskflow to a git wrap; before this the dependency was not fetched via git clone at configure time)

File: subprojects/taskflow.wrap:1-4 (consumed at meson.build:233)

Suggested fix:

1. Preferred: replace the [wrap-git] with a [wrap-file] on the release tarball, mirroring subprojects/liburing.wrap — e.g. source_url = https://github.com/taskflow/taskflow/archive/refs/tags/v3.10.0.tar.gz, plus source_hash, source_fallback_url (wrapdb mirror) and patch_directory = taskflow. Meson's HTTP downloader is what already works in every CI image, and it gains hash verification and cache reuse.
2. Complementary hardening: bake taskflow (header-only) into the .ci/dockerfiles base images or pre-populate subprojects/packagecache/ so configure never needs external network; alternatively add git config --global url."https://github.com/".insteadOf mirroring + GIT_TERMINAL_PROMPT=0 and a retry around meson setup so a refused fetch fails loudly instead of hanging on a credential prompt.
3. Short term: re-run the build — the same clone succeeded in 4 of 6 variants of this very build.

Related: PR #2121 (introduced the taskflow git wrap), PR #2230 (build under test, unaffected by the failure)

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-non-gpu · commit 1d2473cb

TL;DR: The two ubuntu22 build variants failed in meson setup because the taskflow wrap is fetched with an anonymous git clone from github.com, which is answered with a credential prompt ("could not read Username") inside those CI images; switch the wrap to a tarball (wrap-file/source_url) like liburing/tomlplusplus, or pre-install taskflow in the image.

Full analysis

Summary: nixl-ci-non-gpu #3048 — the Build stages for the two Ubuntu 22.04 matrix variants (x86_64 and aarch64, stage IDs 456 and 358) aborted during meson setup nixl_build with ERROR: Subproject taskflow is buildable: NO; the other four variants built and tested fine.

Root cause: meson.build:233 resolves taskflow through the fallback wrap subprojects/taskflow.wrap, which is a [wrap-git] entry pointing at https://github.com/taskflow/taskflow.git. In the nixl-ci-non-gpu-base-ubuntu22 containers that clone fails immediately:

Cloning into 'taskflow'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
meson.build:233:16: ERROR: Git command failed: [... 'clone','--depth','1','--branch','v3.10.0','https://github.com/taskflow/taskflow.git','taskflow']

i.e. the HTTPS git endpoint returned an auth challenge and git had no TTY/credentials. This is environment-specific, not code-specific: in the same build the tarball fetches from GitHub work everywhere (Downloading liburing source from …/liburing-2.14.tar.gz, tomlplusplus), and on the Ubuntu 24.04 images the same git clone succeeds (stage 401: Cloning into 'taskflow' → taskflow| Subproject taskflow finished, plus a successful prometheus-cpp clone). The failure is deterministic (same two variants failed identically in #3047), and it is new — #3044 had all six Build stages green — consistent with the ubuntu22 images being rebuilt (CI_IMAGE_TAG=[REDACTED:Hex High Entropy String], .ci base-image bump) rather than with PR #2230, which only touches Doxygen comments. The structural weakness is that the build hard-requires an unauthenticated git clone at configure time.

Implicated commit: 76275cf "Use tagged Taskflow git wrap for NIXL builds (#2121)" — bzsuni (introduced the [wrap-git] taskflow dependency); the trigger for it surfacing now is the CI base-image rebuild in [REDACTED:Hex High Entropy String] "build: bump CUDA and CI base images…" — NirWolfer.

File: subprojects/taskflow.wrap:1-4 (consumed at meson.build:233)

Suggested fix:

1. Preferred: convert the wrap to a tarball fetch, which is proven to work in all six images:

   [wrap-file]
   directory = taskflow-3.10.0
   source_url = https://github.com/taskflow/taskflow/archive/refs/tags/v3.10.0.tar.gz
   source_filename = taskflow-3.10.0.tar.gz
   source_hash = <sha256 of the tarball>
   patch_directory = taskflow
   [provide]
   taskflow = taskflow_dep
   

   (mirror it on urm.nvidia.com/artifactory if you want to remove the GitHub dependency entirely).
2. Or install taskflow into /opt/nixl in .ci/dockerfiles/Dockerfile.base (as is done for abseil/UCX) so dependency('taskflow') is satisfied by pkg-config/CMake and the fallback is never exercised.
3. Separately, investigate why plain git clone https://github.com/... returns a 401/407 inside the nixl-ci-non-gpu-base-ubuntu22 image (proxy/insteadOf rewrite or missing egress) — the same fragility will hit prometheus-cpp, which is also a git wrap, as soon as those images get past taskflow.
4. Security note, unrelated to the failure: the whole-build console log for #3048 echoes the full githubData webhook payload, which includes an Artifactory basic-auth token. That credential should be rotated and the payload masked in the pipeline (see the earlier "CI: Suppress credentials in logs" work, d6187c6).

Related: PR #2121 (taskflow git wrap), PR #2205 / commit [REDACTED:Hex High Entropy String] (CI base image bump), PR #2230 (the PR under test — not the cause; same failure present in #3047)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id ea34db85-7e2e-432c-b905-f7817a70f52c in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 1d2473cb

TL;DR: All four parallel Build image stages died on git clone https://github.com/... inside the container with fatal: could not read Username for 'https://github.com': No such device or address — GitHub is returning 401/refusing anonymous HTTPS clones from the build hosts, not a code defect in PR #2230. Fix by mirroring/retrying these third-party clones (and failing fast with GIT_TERMINAL_PROMPT=0) rather than relying on unauthenticated github.com fetches at image-build time.

Full analysis

Summary: nixl-ci-build-container-pr #641 failed in every parallel Build image stage (nodes 233, 188, 162, 209) while cloning third-party dependencies (abseil-cpp, grpc + its submodules) during the podman/buildah image build.

Root cause: Unauthenticated git clone against https://github.com from inside the build container got an HTTP 401, so git tried to prompt for credentials and aborted: fatal: could not read Username for 'https://github.com': No such device or address → subprocess exited with status 128 → Error: building at STEP "RUN git clone ...". The failure is environmental and intermittent, not deterministic:

• x86 build (node 233) died at STEP 31/72 on the very first clone (abseil-cpp), 11:37:03.
• arm64 build (node 162) cloned abseil-cpp fine, built and installed it, then died at STEP 32/72 on grpc — the top-level clone and 3 submodules succeeded, then demumble, googletest, protobuf, re2, zlib, boringssl-with-bazel all failed with the same 401 message, including after git's automatic "Retry scheduled" (11:39:47–11:39:50).

That mixed success/failure pattern within a single command is the signature of rate-limiting / a proxy rejecting anonymous git-over-HTTPS, not a bad URL or DNS outage (apt traffic to archive.ubuntu.com / ports.ubuntu.com in the same steps worked fine). Nothing in the PR's branch (tdavidor/device-api-restore-docs, a docs restore) touches these steps.

Implicated commit: unknown — not introduced by [REDACTED:Hex High Entropy String]; the failing RUN git clone steps are long-standing (contrib/Dockerfile last touched by [REDACTED:Hex High Entropy String], NirWolfer, and e6fb771, Adit Ranadive, neither of which changed these lines).

File: contrib/Dockerfile:157 (abseil-cpp clone) and contrib/Dockerfile:174 (grpc --recurse-submodules clone); the same exposure exists at lines 197, 204, 209, 214.

Suggested fix:

1. Immediate: re-run build Added the AIStor plugin support to nixlbench #641 — the failure is transient and the arm64 job proves the same clone succeeds moments apart.
2. Durable, in contrib/Dockerfile:
   • Add ENV GIT_TERMINAL_PROMPT=0 (and GIT_ASKPASS=/bin/true) so a 401 fails immediately with a clear error instead of a confusing "could not read Username".
   • Wrap each clone in a retry loop with backoff, e.g. for i in 1 2 3 4 5; do git clone ... && break || sleep $((i*15)); done, and for grpc use git clone + git submodule update --init --recursive --depth 1 --jobs 4 so submodules can be retried independently.
   • Point these fetches at the internal mirror the way the Ubuntu archive already is (see CI: Switch to the NVIDIA internal Ubuntu mirror #1961 "Switch to the NVIDIA internal Ubuntu mirror"), e.g. git config --global url."<internal-mirror>/".insteadOf "https://github.com/", or fetch pinned release tarballs (ARG ABSL_TAG/GRPC_TAG are already pinned) instead of cloning.
   • Minor efficiency win while you're there: line 157 does a full clone then git fetch --depth 1 origin ${ABSL_TAG}; replace with git clone --depth 1 --branch ${ABSL_TAG}.

Related: none (no existing issue for this clone failure; #1961 is the precedent for moving external fetches to an internal mirror).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id b84c6d13-2afa-42fd-bcf7-6a679f42a5cc in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 1d2473cb

TL;DR: All four parallel container builds fail at the same Dockerfile step — uv pip install torch against https://download.pytorch.org/whl/cu134, an index that has no CPython 3.12 torch wheels — because the index URL is derived from CUDA_VERSION (13.4) after the base image was bumped to cuda13.4. Fix: stop deriving the index from CUDA_VERSION and pin a real PyTorch index (PR #2249 already does this).

Full analysis

Summary: nixl-ci-build-container-pr #642: all 4 parallel "Build image" stages fail at Dockerfile STEP RUN if ... import torch ... else uv pip install --system torch torchvision torchaudio with exit status 1.

Root cause: The Dockerfile computes the PyTorch wheel index from the CUDA version of the base image:
export UV_INDEX="https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d .)".
With BASE_IMAGE_TAG=26.08-cuda13.4-devel-ubuntu24.04, CUDA_VERSION=13.4 yields cu134 — an index PyTorch does not publish current wheels for. The log shows uv's resolution failing:

• error: No solution found when resolving dependencies
• cause: Because all versions of torch have no wheels with a matching Python ABI tag (e.g., `cp312`)
• hint: `torch` was found on https://download.pytorch.org/whl/cu134, but not at the requested version
• hint: You require CPython 3.12 (`cp312`), but we only found wheels for `torch` (v2.0.1) with ... `cp38`, `cp39`, `cp310`, `cp311`

The if probe for a pre-installed torch ≥2.7 also fails on this base image (no system torch), so the fallback branch always runs. This is a build-config defect, not infra and not related to the PR's doc changes on branch tdavidor/device-api-restore-docs — the same failure reproduces in every parallel variant at the identical step.

Implicated commit: [REDACTED:Hex High Entropy String] — "build: bump CUDA and CI base images, stop restating them across CI (#2205)", NirWolfer (bumped BASE_IMAGE_TAG/CUOBJ_DEV_IMAGE to cuda13.4, making the derived cu134 index invalid).

File: contrib/Dockerfile:333-338 (index derivation on line 336)

Suggested fix: Stop deriving the index from CUDA_VERSION. Introduce an explicit build arg, e.g. ARG TORCH_INDEX_URL="https://download.pytorch.org/whl/cu130", and use it directly:

export UV_INDEX="${TORCH_INDEX_URL}" && \
uv pip install --system torch torchvision torchaudio

Optionally add --index-strategy unsafe-best-match or a PyPI fallback so a missing CUDA-specific index degrades gracefully instead of resolving to torch 2.0.1. PR #2249 ("build: pin the torch wheel index instead of deriving it from CUDA_VERSION") is exactly this change — merge/rebase onto it and re-run this PR's container build.

Related: PR #2249 (pin the torch wheel index instead of deriving it from CUDA_VERSION); PR #2205 / commit [REDACTED:Hex High Entropy String] (CUDA base image bump that triggered it).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 415822aa-dbe8-4443-91e6-36a6df46182d in the triage console for the audit trail.

[tomerdav]

/build

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu · commit 92197258

TL;DR: test_prep_mem_view fails on the GB200 DL nodes because UCX 1.23's ucp_device_local_mem_list_create rejects the memh NIXL passes (invalid memh for md_index=6/7), so prepMemView returns NIXL_ERR_BACKEND; this is a pre-existing device-API/UCX-1.23 incompatibility, not caused by the docs-only PR #2230, and needs the local mem-list registration fixed (or the test gated) rather than a re-run.

Full analysis

Summary: Stage "Run DL Python tests" (node gb200-nvl4-ts2-65, slurm job 2207991) failed: pytest -s test/python → FAILED test/python/test_nixl_api.py::test_prep_mem_view (1 failed, 25 passed, 2 skipped).

Root cause: Both spawned ranks die inside the local overload of prep_mem_view:

ucp_device.c:249  UCX ERROR invalid memh for md_index=6      (rank 0)
ucp_device.c:341  failed to pack local mem list element for element=0
ucp_device.c:466  failed to create local mem list handle: Invalid parameter
ucx_backend.cpp:796] Failed to prepare local memory view: Failed to create device memory list(local): Invalid parameter
→ nixl_cu13._bindings.nixlBackendError: NIXL_ERR_BACKEND (test_nixl_api.py:293)

nixl::ucx::createMemList(const nixl_meta_dlist_t&, ...) builds each element from the private metadata's memh (element.memh = md->getMem().getMemh()) and passes the shared worker to ucp_device_local_mem_list_create. The DL container ships UCX 1.23.0 (from the build log: Run-time dependency ucx found: YES 1.23.0, UCX GPU Device API: YES), whose device path requires the memh to carry a registration on the md used by the worker's device (GDA/cuda_ipc) lane. NIXL registers VRAM with a plain ucp_mem_map (flags = 0, no memory-type/device hint) in nixlUcxContext::memReg, so the handle has no registration for md_index 6/7 → UCS_ERR_INVALID_PARAM.

This is not related to the commit under test: PR #2230 is "Restore user-facing Device API Doxygen comments" (comments only). Build #2302 (different commit, node gb200-nvl4-ts2-102) fails with the identical error and md indices, i.e. it reproduces whenever the test actually runs (≥2 GPUs + HAVE_UCX_GPU_DEVICE_API); #2303 passed only because the test did not exercise this path. No hang is involved — the log is continuous and the run finished in 24.8 s.

Implicated commit: Not the PR commit. The failing code paths come from 0c654f24 (Raul Akhmetshin, "PLUGINS/UCX: Device API V2. Backend implementation." #1245) and the test/binding added in [REDACTED:Hex High Entropy String] (x41lakazam, "BINDINGS/PYTHON: Expose prepMemView (local + remote overloads)" #1715), combined with the container's UCX 1.23 upgrade.

File: src/plugins/ucx/mem_list.cpp:110-122 and :183-195 (local element/mem-list creation); src/plugins/ucx/ucx_utils.cpp:646-662 (memReg, ucp_mem_map without memory-type/device flags); surfaced at src/plugins/ucx/ucx_backend.cpp:796 and test/python/test_nixl_api.py:293.

Suggested fix:

1. Real fix — make the VRAM registration device-capable under UCX ≥1.23: in nixlUcxContext::memReg set UCP_MEM_MAP_PARAM_FIELD_MEMORY_TYPE (UCS_MEMORY_TYPE_CUDA for VRAM_SEG) and the UCX 1.23 device/export map flag when HAVE_UCX_GPU_DEVICE_API, and verify in createMemList that the memh belongs to the same worker/context whose device lane is selected (currently createElements is called with the default worker_id = 0 for the local path while prepMemView picks getSharedWorkerId(opt_args)). Confirm against the UCX 1.23 ucp_device_local_mem_list_create requirements and cover it with the existing gtest device-API tests.
2. CI unblock in the meantime — since the failure is independent of PR device api: Restore user-facing Device API Doxygen comments #2230, either add a UCX-version guard (xfail/skipif for UCX ≥ 1.23 until the registration fix lands) to test_prep_mem_view, or pin the DL container back to the UCX version this test was validated against; do not just retry the build.

Related: PR #2230 (branch under test, docs-only); PR #2194 "bugfix: test: make the device API tests actually run"; prior device-API PRs #1715, #1245; also seen in build #2302 on gb200-nvl4-ts2-102 (slurm job 2207509).

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 83c318ed-dcf5-4bd6-9845-0f4e2c6c7b8c in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-container-pr · commit 92197258

TL;DR: All four container-image builds hit transient unauthenticated git clone failures against github.com (fatal: could not read Username for 'https://github.com'), which is an infrastructure/network problem unrelated to this docs-only PR; fix by adding retries + mirroring/authenticating the third-party clones in contrib/Dockerfile (and making the UCX GPUNetIO submodule failure fatal rather than a warning).

Full analysis

Summary: Three of the four parallel "Build image" stages in nixl-ci-build-container-pr #685 failed at different git clone steps inside the container build; the fourth (id 233) survived and took 46 min.

Root cause: Every failure is the same signature — git got an HTTP 401 from github.com and tried to prompt for credentials with no tty:

• Stage 198 (x86, step 32/73): Cloning into 'abseil-cpp'... fatal: could not read Username for 'https://github.com': No such device or address → exit status 128
• Stage 180 (arm64, step 33/73): abseil and most of grpc's ~30 submodules cloned fine, then Unable to fetch in submodule path 'third_party/protoc-gen-validate' → same could not read Username → exit status 128
• Stage 209 (UCX build stage): Cloning into '/workspace/ucx/external/gpunetio'... fatal: could not read Username twice, then WARNING: Failed to update GPUNetIO submodule, continuing.... The build proceeded anyway and died later at make[5]: *** No rule to make target 'gpunetio/common/doca_gpunetio_verbs_def.h', needed by 'all-am' → exit status 2.

The partial-success pattern (same URL succeeding and failing minutes apart in the same build) rules out a bad URL or a missing tag and points to github.com throttling/refusing anonymous requests. Four parallel arch/variant image builds each perform dozens of credential-less clones (grpc's recursive submodules alone are ~30 repos, several fetched twice) from one egress IP. This is not a code defect: PR #2230 is a docs-only Doxygen-comment change, and none of the failing steps touch changed files.

A secondary, real bug is exposed by stage 209: UCX's autogen.sh downgrades the failed GPUNetIO submodule clone to WARNING: ... continuing..., but configure still enables MLX5 modules: < gda >, so make fails ~40 s later on a header that submodule was supposed to provide. That turns a retryable network blip into a confusing compile error.

Implicated commit: none — no commit in the repo caused this. Nearest related change is 029a8544 (Alexey Rivkin, "CI: Switch to the NVIDIA internal Ubuntu mirror"), which mirrored apt but left the github.com clones direct.

File: contrib/Dockerfile:161 (abseil clone), contrib/Dockerfile:178 (grpc --recurse-submodules clone), and the UCX build stage that runs ./autogen.sh

Suggested fix:

1. Re-run the build first — this is transient and stage 233 proves the same Dockerfile succeeds.
2. Harden the clones in contrib/Dockerfile: export GIT_TERMINAL_PROMPT=0 so a 401 fails fast instead of hanging on a prompt, and wrap each git clone/submodule update in a retry loop with backoff (the DOCA wget on line 135 already does this with --tries=3 --waitretry=5; the clones have no equivalent).
3. Reduce anonymous request volume: use --depth 1 for abseil (line 161 clones full history then git fetch --depth 1), and consider -c url.https://github.com/.insteadOf pointing at an internal GitHub mirror/cache, following the precedent of the internal Ubuntu mirror. Alternatively inject a GitHub token via a build secret so clones are authenticated and get the much higher rate limit.
4. Make the UCX GPUNetIO submodule failure deterministic: either fail the step when the submodule clone fails, or pass --without-gdaki (disable the gda MLX5 module) when external/gpunetio is absent, so the build doesn't proceed into a guaranteed No rule to make target ... doca_gpunetio_verbs_def.h.

Related: PR #2230 (the PR under test — docs only, not the cause). Same-signature clone flakiness is plausible in the other currently-open container-build PRs #2250, #2231.