[NirWolfer]

Summary

The Artifactory cleanup job (nixl-ci-cleanup-artifacts) has failed since the move to the IL instance with This endpoint does not support reference tokens as the bearer credential. jf config add with --user/--password enables token auto-refresh by default, so the first jf command calls POST api/security/token to mint a refreshable token, and that endpoint rejects the reference token in svc-nixl-il-artifactory-token. The token itself is fine: the other jobs use it through docker login and curl basic auth, which the IL instance accepts.

Changes

• artifactory-cleanup-by-time.sh configures jf with --basic-auth-only, which skips the token exchange and sends the credential as basic auth
• ci-overview.md notes why the flag is there

Summary by CodeRabbit

• Documentation
  • Clarified that cleanup uses basic authentication because the configured reference token is incompatible with JFrog’s automatic token refresh.
• Chores
  • Updated artifact cleanup authentication settings to skip JFrog’s token exchange endpoint.

[github-actions]

👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[NirWolfer]

Dry run test on the cleanup pipeline for this PR: https://nbuprod.blsm.nvidia.com/nbu-swx-nixl-main/job/Devops/job/nixl-ci-cleanup-artifacts/161/

[NirWolfer]

/build

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/nixl/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: b461e8a0-e790-4b92-8164-cf498be5ef92

📥 Commits

Reviewing files that changed from the base of the PR and between 13a896b and c8659a1.

📒 Files selected for processing: 2

• .ci/docs/ci-overview.md
• .ci/scripts/artifactory-cleanup-by-time.sh

📝 Walkthrough

The Artifactory cleanup script now configures jf to use basic authentication only. The CI overview explains that this avoids token auto-refresh rejecting the IL reference token.

Changes: Artifactory cleanup authentication

Layer / File(s)	Summary
Configure and document basic authentication .ci/scripts/artifactory-cleanup-by-time.sh, .ci/docs/ci-overview.md	The script adds --basic-auth-only and notes that it skips JFrog’s token exchange endpoint. The CI overview documents why the option is used with the IL reference token.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Bug fix

Suggested reviewers: ntsemah

Merge Risk: ⚪ Minimal · up to c8659

The cleanup job now avoids the reported token-refresh failure path using a supported CLI option. Acceptance of live IL credentials remains unverified.

Pre-merge checks: 5

Check name	Status	Explanation
Description check		The description explains what changed, why the change is required, and how the fix avoids the rejected token exchange. The content is complete even though it uses Summary and Changes headings instead …
Title check		The title clearly and concisely describes the primary change: using basic authentication for JFrog in the Artifactory cleanup job.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains what changed, why the change is required, and how the fix avoids the rejected token exchange. The content is complete even though it uses Summary and Changes headings instead of the template headings.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-gpu · commit c8659a15

TL;DR: Build #4197 failed because GPU hardware on the allocated node mizu01 was in a broken PCIe/driver state — nvidia-container-cli could not initialize NVML, so pyxis could never start the test container. This is an infrastructure fault unrelated to the PR's artifactory basic-auth change; drain mizu01 for a reboot and re-run.

Full analysis

Summary: The "Sync Container Image" stage failed when pyxis/enroot could not start the container on mizu01: nvidia-container-cli: detection error: nvml error: unknown error, causing srun task 0 to exit 1.

Root cause: GPU/PCIe hardware fault on the allocated node, not a code or CI-config defect. Sequence from the logs:

• The image pulled and imported from artifactory successfully (pyxis: imported docker image: ...nixl-ci-gpu-test:4197 at 13:35:25), so the PR's basic-auth change worked as intended — credentials were not the failure point.
• Container start then failed at 13:36:46 in enroot's GPU hook: /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1, preceded by nvidia-container-cli: detection error: nvml error: unknown error. That hook's only job is to enumerate GPUs via NVML and inject them into the container; an "unknown error" from NVML means the driver could not talk to the devices at all.
• logs.syslog_logs_local for host mizu01 corroborates a device falling off the bus in exactly this window:
  • 13:05:01 kernel: mlx5_core 0000:4b:00.0: not ready 8191ms after bus reset; waiting
  • 13:22:05 kernel (err): pcieport 0000:51:00.0: Unable to change power state from D3cold to D0, device inaccessible — 14 minutes before the container-start failure
  • 13:39:51 srp_daemon (err): Device has not yet recovered from catas error
• Slurm job records confirm allocation 109750 (nixl-ci-gpu-4197) ran on mizu01, started 13:33:27 and terminated 13:41:37 in state CANCELLED with exit_code 0:0 and no state_reason — i.e. torn down by the pipeline after the step failed, not killed by slurm for time or memory. There was no competing tenant: the only other job touching the node (109590 / build 4196) had already ended at 12:24:35.

This is not a timeout or a hang — the stage's largest gap is the ~2 min image import, which completed normally, and the failure is an immediate non-zero exit.

Implicated commit: none — [REDACTED:Hex High Entropy String] (branch ci/artifactory-cleanup-basic-auth, PR #2447) is not implicated; the artifactory pull it changes succeeded in this very log.

File: n/a — failure is in node-side /etc/enroot/hooks.d/98-nvidia.sh on mizu01, triggered by host PCIe state, not by any file in this repo.

Suggested fix:

1. Immediate: Re-run build #4197. Since mizu01 is still suspect, either exclude it (salloc ... --exclude=mizu01) or allocate with --comment='reboot-before' — the salloc banner in the "Allocate Environment" stage advertises exactly this option, and a reboot is the standard recovery for a device stuck in D3cold.
2. Node remediation (file with infra): scontrol update NodeName=mizu01 State=DRAIN Reason="PCIe 0000:51:00.0 inaccessible from D3cold; NVML init failure" and power-cycle the host. Quote slurm allocation 109750 and the 13:22:05 kernel error — that's what an admin needs to dig further. A warm reboot may not clear a D3cold transition failure; a full power cycle is more reliable.
3. CI hardening (worth doing regardless): the GPU health probe in the pipeline (nvidia-smi -L || echo no-gpu-visible) runs inside the container, so it can never detect this class of fault — the container fails to start before the probe executes. Add a pre-flight nvidia-smi -L on the host right after allocation and fail fast / re-queue onto another node when it errors. That converts a confusing mid-pipeline container error into an actionable "node unhealthy" signal, and would let the job retry on a healthy node automatically instead of reporting a red build against an innocent PR.

Related: No matching issues or PRs found in the nixl repo for this signature; worth opening an infra ticket for mizu01 rather than a code issue.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id be392dfd-59b7-44f8-9af2-b0c3bd385f31 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-dl-gpu-ep · commit c8659a15

TL;DR: The third elastic-EP case (no_expansion.json --disable-ll-nvlink, i.e. the RDMA path) failed because UCX could not build a device memory list — "lane not found for element 1" / "failed to pack uct memory element" — so all 4 ranks aborted; the test script unsets UCX_NET_DEVICES and leaves UCX_TLS unpinned for the RDMA case, letting UCX auto-select non-GDA-capable lanes. Pin the transports/devices for that case (exactly what open PR #2430 does); this is unrelated to the branch's Artifactory-auth commit.

Full analysis

Summary: Stage "Run DL NIXL EP tests" (node 207) failed when examples/device/ep/tests/elastic/elastic.py --plan no_expansion.json --disable-ll-nvlink aborted with all four worker processes exiting 1.

Root cause: The first two (NVLink) elastic plans passed. The RDMA variant failed immediately at connection setup on gb-nvl-118-compute01:

ucp_device.c:669  UCX ERROR failed to pack uct memory element for first element
ucp_device.c:700  UCX ERROR lane not found for element 1
ucp_device.c:819  UCX ERROR failed to create handle: Invalid parameter
ucx_backend.cpp:809] Failed to prepare remote memory view: Failed to create device memory list(remote): Invalid parameter
...
RuntimeError: Worker processes failed: worker 0..3 (exit code 1)

This is a UCX transport/device-selection problem, not a hang (the whole run took ~55 s with continuous output). In .gitlab/test_ep.sh:71-76, run_elastic_test unsets UCX_NET_DEVICES ("let UCX auto-select GPU-capable transport") and only exports UCX_TLS=^rc_gda for the non---disable-ll-nvlink cases; for the RDMA case neither variable is constrained. UCX then auto-selects a lane set that has no GDA-capable lane for the remote memory element — note the node also exposes mlx5_4/mlx5_5 in PORT_DOWN (Ethernet) alongside the four active CX-7 HCAs, while the all_rdma_nics_active gate (lines 95-103) only verifies mlx5_0..mlx5_3, so the RDMA block ran regardless. nixlUcxEngine::prepMemViewImpl catches the exception from nixl::ucx::createMemList and returns NIXL_ERR_BACKEND, which propagates as the worker failure.

Nothing in this failure touches the PR's subject (Artifactory basic auth); the build's Docker/compile/sync stages all succeeded and the failure is in the RDMA EP test path.

Implicated commit: Not caused by [REDACTED:Hex High Entropy String]. The relevant test-harness logic dates to [REDACTED:Hex High Entropy String] (Itay Alroy, "nixl_ep: Test vLLM Elastic EP without patching vLLM", #2300) and earlier 55bcf4dc (lishapira, "ci: Add nixl ep to new dl ci pipeline", #1802).

File: .gitlab/test_ep.sh:71-76 (device/TLS unsetting in run_elastic_test); error surfaced at src/plugins/ucx/ucx_backend.cpp:809

Suggested fix: Stop relying on UCX auto-selection for the RDMA case — pin both the transport and the HCAs in run_elastic_test, e.g. for the --disable-ll-nvlink branch export a GDA-capable UCX_TLS (including rc_gda) and UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1 so the down CX-8 Ethernet ports (mlx5_4/mlx5_5) can never be chosen. Open PR #2430 ("ci: pin UCX transports for the RDMA elastic EP tests") implements precisely this — land it and re-run #1813. Do not treat this build as a signal against the ci/artifactory-cleanup-basic-auth change.

Related: PR #2430 (ci: pin UCX transports for the RDMA elastic EP tests); PR #2248 (Nixl ep ci extend tests coverage)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 3a412444-bc77-42dd-80ff-a949eadb2bb1 in the triage console for the audit trail.

[svc-nixl]

🤖 CI Triage Agent — nixl-ci-build-wheel · commit c8659a15

TL;DR: The SGLang sanity test failed because the SGLang prefill server could not bind its assigned HTTP port 21601 ("[Errno 98] address already in use"), died, and the script's check_alive aborted the stage; the port had been handed out ~2 minutes earlier by get_next_tcp_port() and was taken by something else on the shared host network in between. Fix the port handout: derive CI_CONCURRENT_ID deterministically/uniquely (e.g. from the Jenkins build number) instead of $RANDOM % 45, and re-check/retry the port immediately at server launch.

Full analysis

Summary: Stage "Run SGLang sanity" (node 680) failed — srun: error: gb-nvl-082-compute09: task 0: Exited with exit code 1 after the SGLang prefill instance failed to bind port 21601 and was killed.

Root cause: Port collision on the host-networked GPU node, not a NIXL code defect.

• .ci/scripts/common.sh computed nixl_concurrent_id=37 → port block 21600–21750, so get_next_tcp_port() handed out PREFILL_PORT=21601, DECODE_PORT=21602, PROXY=21603, BOOTSTRAP=21604, NCCL 21605/21606 (all confirmed in the server_args dumps).
• At 06:51:55 the ports were verified free. The prefill engine then spent ~60 s loading weights and capturing CUDA graphs, and only at 06:53:52 did uvicorn try to bind: ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 21601): address already in use. Its own warmup then hit HTTPConnectionPool(host='127.0.0.1', port=21601) ... Connection refused, kill_process_tree called: parent_pid=2811612, [sanity:sglang] prefill (pid 2811612) exited. The decode instance on 21602 came up fine, so the clash is specific to 21601.
• Two design flaws make this inevitable rather than unlucky: (1) get_next_tcp_port() is a check-then-use race — the free-port probe happens minutes before the server actually binds (.ci/scripts/common.sh:50); (2) the concurrency id is $((RANDOM % 45)) and the port file /tmp/nixl_tcp_port_<id> lives inside the per-build container, so every job with the same random id deterministically starts at the same base port (tcp_port_min + 1) on a node whose network namespace is shared with every other enroot container. Two sanity/CI jobs landing on the same node with the same id collide exactly like this.
• Unrelated to the PR's change set (branch ci/artifactory-cleanup-basic-auth touches Artifactory auth); the vLLM sanity stage on the other node passed. This is a CI flake.

Implicated commit: 21764ff6 — NirWolfer, "CI: pin SGLang rendezvous ports and fail fast on dead servers in sanity" (#2359), which introduced the pinned $RANDOM-based CI_CONCURRENT_ID fallback; the underlying TOCTOU port handout dates to f125c7bc (Alexey Rivkin, #1685).

File: .gitlab/test_vllm_sglang_sanity.sh:33 (random CI_CONCURRENT_ID) and .ci/scripts/common.sh:38-61 (get_next_tcp_port)

Suggested fix:

1. Make the port block collision-free instead of probabilistic: export a deterministic CI_CONCURRENT_ID into the srun environment from the pipeline (e.g. BUILD_NUMBER % tcp_port_slots, plus a distinct offset for the vllm vs sglang variant) so two jobs on the same node cannot share a block. Jenkins already knows the build number — it is in the container name nixl-vs-sglang-2302.
2. Close the TOCTOU: wrap the SGLang/vLLM launch in a retry that, on server exit with address already in use, re-runs get_next_tcp_port() and relaunches (2–3 attempts). Cheaper alternative: re-validate the port with ss -tuln immediately before setsid ... launch_server and skip forward if taken.
3. Harden the probe itself — grep -q :$next_port is an unanchored substring match and can both skip free ports and miss matches; use ss -Htuln | awk '{print $5}' | grep -qE "[:.]${next_port}$".
4. Re-run build [GPUNetIO] Select the engine GPU before QP allocation #2302 to confirm it is a flake.

Related: #2359, #1995 (prior SGLang router port-collision fix in the same test) — this is a recurrence of the same class of problem.