ci: authenticate HuggingFace model prefetch to avoid rate limits - #1948
Conversation
|
👋 Hi NirWolfer! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughJenkins configures an internal Hugging Face mirror for vLLM and SGLang sanity containers. The prefetch script sets longer Hugging Face Hub timeouts, and CI documentation describes the endpoint propagation. ChangesHugging Face sanity prefetch
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.ci/jenkins/lib/build-wheel-matrix.yaml:
- Around line 79-80: Restrict the HF_TOKEN binding from the top-level wheel job
credentials to only the vLLM/SGLang sanity steps in the matrix. Update the
relevant sanity-step wrapper or invocation so the token is available there while
remaining unavailable to build and image stages that execute PR-controlled code.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 0320dcd5-72f6-4a52-9d9e-f9444166d13c
📒 Files selected for processing (3)
.ci/docs/ci-overview.md.ci/jenkins/lib/build-wheel-matrix.yaml.gitlab/test_vllm_sglang_sanity.sh
bd426fa to
d3d8f6e
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
.ci/docs/ci-overview.md (2)
30-35: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCorrect the Jenkins job count.
This note says there are 13 jobs (the dispatcher, 7 children, and 5 standalone jobs), but Line 140 still says 12. Update the later count so the overview is consistent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.ci/docs/ci-overview.md around lines 30 - 35, Update the Jenkins job count in the later overview section to 13, keeping it consistent with the breakdown documented in the note and leaving the job descriptions unchanged.
170-170: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd blank lines before these headings.
Insert a blank line before the headings on Line 170 and Line 175 to satisfy Markdownlint MD022.
Also applies to: 175-175
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.ci/docs/ci-overview.md at line 170, Insert a blank line immediately before the `nixl-ci-build-container-pr` heading and the other heading at the referenced later location, preserving their content and satisfying Markdownlint MD022.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.ci/docs/ci-overview.md:
- Around line 30-35: Update the Jenkins job count in the later overview section
to 13, keeping it consistent with the breakdown documented in the note and
leaving the job descriptions unchanged.
- Line 170: Insert a blank line immediately before the
`nixl-ci-build-container-pr` heading and the other heading at the referenced
later location, preserving their content and satisfying Markdownlint MD022.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 0bebbed7-237c-4854-beee-cb8f01f91dc1
📒 Files selected for processing (3)
.ci/docs/ci-overview.md.ci/jenkins/lib/build-wheel-matrix.yaml.gitlab/test_vllm_sglang_sanity.sh
|
/build |
d3d8f6e to
687b5ad
Compare
|
/build |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.ci/jenkins/lib/build-wheel-matrix.yaml:
- Around line 219-235: Remove the pipeline_start credential binding and global
env.HF_TOKEN assignment. Scope both credentials inside the vLLM/SGLang
sanity-step invocations, forwarding HF_ENDPOINT and HF_TOKEN only to those
commands while leaving later build scripts and Docker builds without token
access.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 8e810bf4-44b9-4966-b06c-d142f02479bb
📒 Files selected for processing (3)
.ci/docs/ci-overview.md.ci/jenkins/lib/build-wheel-matrix.yaml.gitlab/test_vllm_sglang_sanity.sh
|
🤖 CI Triage Agent — TL;DR: The SGLang sanity job failed because the model prefetch ( Full analysisSummary: Stage "Run SGLang sanity" (node 534) failed during model weight prefetch, not during NIXL/build logic. Root cause: At 14:02:47 the script started Implicated commit: unknown — this is an environmental failure, not a code regression. (The prefetch script File: Suggested fix:
Security note: The Related: none found. |
687b5ad to
c38742e
Compare
|
/build |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.gitlab/test_vllm_sglang_sanity.sh:
- Around line 104-105: Update the HF_HUB_ETAG_TIMEOUT and
HF_HUB_DOWNLOAD_TIMEOUT defaults in the sanity-job environment setup to a
substantially smaller value bounded by the CI job budget, while preserving any
explicitly provided environment values and leaving retry behavior unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: b13a6a0b-126e-4adc-b128-cad8232c9715
📒 Files selected for processing (3)
.ci/docs/ci-overview.md.ci/jenkins/lib/build-wheel-matrix.yaml.gitlab/test_vllm_sglang_sanity.sh
|
🤖 CI Triage Agent — TL;DR: The "Test CPP" stage failed because Full analysisSummary: Jenkins stage "Test CPP" (node 462, x86_64/nixl-base-25.06-cuda12.9-ubuntu24.04) failed with exit code 245 because the gtest binary reported Root cause: A segmentation fault inside the UCX backend during the Implicated commit: unknown — the crash is in the UCX backend disconnect path exercised by File: Suggested fix: This is a backend crash unrelated to PR #1948, so re-running CI should unblock the PR. To fix the underlying flake, reproduce locally with Related: PR #1948 (unrelated, HuggingFace CI auth); PR #1743 (run gtest in single process — may reduce cross-test interference); PR #1709 (ASan/UBSan/TSan builds, useful to root-cause this SIGSEGV). |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 220 "Run DL EP elastic tests" (UCX v1.22.x path) failed on plan Root cause: In phase 2 the plan Implicated commit: Not definitively pinpointable from logs alone. The most likely-related change is the low-latency NVLink / CUDA-IPC path work — File: Suggested fix: Make the EP dispatch/combine path fault-safe when a peer rank disappears mid-transfer on the UCX 1.22.x runtime: guard CUDA-IPC endpoint/stream/event destruction so a dead peer's mapping cannot cause an illegal memory access (e.g., validate/clear the peer's IPC handles before Related: #1751 (LL NVLink path), #1846 (forcibly close UCP endpoints); no existing issue matches the "illegal memory access" elastic-EP crash — consider filing one.
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 337 "Run Nixlbench tests" failed because a Root cause: A segfault in the UCCL backend on the shutdown path. The log shows the benchmark completed and printed its result row (
This is a crash, not a wall-clock/hang issue — the log has continuous activity right up to the fault (no multi-minute gap). Note also that every UCCL run reports pathological bandwidth (~0.03 GB/s vs 2–5 GB/s for UCX) and the fault occurs during Implicated commit: Not introduced by this PR. UCCL plugin last changed in File: Suggested fix: Land the UCCL crash fix from PR #1724 (fixes the SIGSEGV in UCCL engine teardown). As an immediate CI unblock, skip/quarantine the UCCL nixlbench backend cases (the loop that runs Related: PR #1724 "UCCL crash Fix + GDB trap" (#1724); possibly related shutdown-race fix PR #1963 "NIXLBENCH: Fix race condition with ASIO runtime on shutdown" (merged 2026-07-21, #1963).
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Elastic EP fault-injection test ( Root cause: The plan intentionally kills rank 2 during phase-2 dispatch/combine ("[rank 2] Killing rank during dispatch/combine", "SIGTERM (15) received for process 51914"). When rank 2 dies, its CUDA-IPC / NVLINK-mapped peer memory becomes invalid. The surviving ranks (0, 1, 3) then hit Implicated commit: Most likely File: Suggested fix: Handle peer-kill CUDA-IPC teardown gracefully so surviving ranks don't dereference invalidated IPC memory: on detecting a removed/killed peer, close/unmap that peer's CUDA-IPC endpoint and skip Related: PR #1846 (UCP forcible endpoint close), PR #1099 (CUDA IPC NVLINK backend), PR #1500 (planned SIGTERM handling in elastic test), PR #1953 (UCX version logging). No existing issue found for this exact illegal-memory-access-on-peer-kill signature — worth filing one.
|
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage 220 "Run DL EP elastic tests" (ucx-master variant) failed; the fault-tolerance plan Root cause: In phase 2 the test expects the ht dispatch/combine kernel to detect the killed peer, hit the configurable GPU timeout ( Implicated commit: [REDACTED:Hex High Entropy String] — "EXAMPLES/DEVICE/EP/CSRC: Removed extra params from ht functions. (#1812)", Raul Akhmetshin, 2026-07-22 (only recent change to File: Suggested fix: Review commit #1812: confirm the removed parameters to the ht dispatch/combine functions did not drop the peer/timeout/mask-status argument that lets a surviving rank bound its access to a peer that may have died. The ht kernel must bounds-check/guard remote cuda_ipc pointers and honor the GPU timeout so it returns and marks the peer failed instead of accessing freed IPC memory. As an interim step, run the elastic test against the exact ucx-master SHA to confirm the UCX-side Related: PR #1812 (commit [REDACTED:Hex High Entropy String]); triggering PR #1948. Search the tracker for "elastic EP illegal memory access fault contraction" to correlate.
|
|
🤖 CI Triage Agent — TL;DR: The build and Docker image compilation all succeeded; the job failed in the "Allocate DL EP Environment" stage because the SLURM Full analysisSummary: Jenkins stage "Allocate DL EP Environment" (node 183/200) failed after ~60 min when SLURM could not allocate a GB200 test node. Root cause: Implicated commit: none — not caused by commit [REDACTED:Hex High Entropy String] or PR #1948; infrastructure failure. File: n/a (Jenkins pipeline SLURM allocation step, Suggested fix: Re-run the build — this is a transient GB200 cluster-capacity/connectivity failure. If it recurs, (1) check Related: none
|
The vLLM/SGLang sanity stages fetch Qwen/Qwen3-8B at runtime from huggingface.co, which rate-limits the shared CI egress IP (429), failing the prefetch before the servers even start. Route the prefetch through the internal urm HuggingFace mirror instead: SANITY_HF_ENDPOINT is passed into the SLURM container as HF_ENDPOINT, where huggingface_hub picks it up. The mirror allows anonymous reads, so no token is involved. Long HF_HUB_*_TIMEOUT values absorb the mirror's cold-cache first-fetch latency. Signed-off-by: NirWolfer <nwolfer@nvidia.com>
c38742e to
90a98e4
Compare
|
/build |
There was a problem hiding this comment.
♻️ Duplicate comments (1)
.gitlab/test_vllm_sglang_sanity.sh (1)
105-106:⚠️ Potential issue | 🟠 MajorBound Hugging Face timeouts to the CI budget.
The 86,400-second defaults can outlive the 40-minute Jenkins sanity timeout, so a stalled request may hold the job and allocated resources until the outer timeout kills it. This repeats the previously reported timeout-budget issue; preserve explicit overrides but use a bounded default such as 900–1800 seconds.
Proposed fix
-export HF_HUB_ETAG_TIMEOUT="${HF_HUB_ETAG_TIMEOUT:-86400}" -export HF_HUB_DOWNLOAD_TIMEOUT="${HF_HUB_DOWNLOAD_TIMEOUT:-86400}" +export HF_HUB_ETAG_TIMEOUT="${HF_HUB_ETAG_TIMEOUT:-1800}" +export HF_HUB_DOWNLOAD_TIMEOUT="${HF_HUB_DOWNLOAD_TIMEOUT:-1800}"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.gitlab/test_vllm_sglang_sanity.sh around lines 105 - 106, Reduce the default values for HF_HUB_ETAG_TIMEOUT and HF_HUB_DOWNLOAD_TIMEOUT in the sanity script to a bounded 900–1800 second range so requests remain within the CI budget, while preserving explicit environment variable overrides.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In @.gitlab/test_vllm_sglang_sanity.sh:
- Around line 105-106: Reduce the default values for HF_HUB_ETAG_TIMEOUT and
HF_HUB_DOWNLOAD_TIMEOUT in the sanity script to a bounded 900–1800 second range
so requests remain within the CI budget, while preserving explicit environment
variable overrides.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: da4872f4-d746-4bb8-90ad-b884e0048f46
📒 Files selected for processing (3)
.ci/docs/ci-overview.md.ci/jenkins/lib/build-wheel-matrix.yaml.gitlab/test_vllm_sglang_sanity.sh
93c7d80 to
90a98e4
Compare
|
/build |
|
🤖 CI Triage Agent — TL;DR: The "Run Nixlbench tests" stage crashed with a SIGSEGV (exit code 139) inside the UCCL backend during engine teardown of the Full analysisSummary: Jenkins Root cause: A segmentation fault in the UCCL backend. The benchmark itself completed — the log shows the final result line printed ( Implicated commit: unknown for the exact regression; the UCCL backend is owned by Pravein Govindan Kannan and the most recent substantive UCCL changes are File: Suggested fix: Reproduce locally with the exact failing config ( Related: PR #1948 (triggering, CI-only, unrelated); UCCL backend PRs #1428, #1271, #895. |
|
🤖 CI Triage Agent — TL;DR: The Full analysisSummary: Stage "Allocate DL EP Environment" (node 183/200) failed with Root cause: GPU resource allocation on the Slurm cluster timed out. The log shows a single blocking Implicated commit: unknown — not caused by commit 66609d9; the failure is in CI infrastructure/cluster scheduling, not the built code (build + install + image push all succeeded). File: N/A (CI pipeline Slurm allocation step: Suggested fix: Retrigger the build once GB200 nodes on the Related: none |
3 participants
What?
The shared CI egress was being rate-limited by HuggingFace on anonymous requests, causing the vLLM and SGLang sanity stages to fail during Qwen/Qwen3-8B prefetch.
Bind the svc-nixl-hf-token Jenkins secret text credential as HF_TOKEN in the build-wheel pipeline and pass it through to the sanity testScript so snapshot_download authenticates rather than hitting the anonymous rate limit.
Why?
rate limit failing build-wheel pipeline on rare occasions
How?
It is optional, but for complex PRs, please provide information about the design,
architecture, approach, etc.
Summary by CodeRabbit
Bug Fixes
Documentation
huggingface.co) and how the mirror endpoint is supplied to the SLURM sanity container.