Skip to content

[Fix] Skip HF Hub API calls for cached repos and honor --revision - #43092

Open
mmangkad wants to merge 11 commits into
mainfrom
mmangkad/ci-hf-hub-rate-limit
Open

mmangkad wants to merge 11 commits into
mainfrom
mmangkad/ci-hf-hub-rate-limit

Conversation

@mmangkad

@mmangkad mmangkad commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Loading a cached model still calls the HF Hub API: transformers' revision lookups (new in 5.18) and repo listings, sglang's own checks, and dataset loads. In CI this exhausts the shared api rate limit (429). Several loads also ignored --revision and read main.

Merged main after #43229 landed; conflicts resolved as follows:

In main Used Why
Engine offline hook Main, plus this PR's revision and draft handling Covers in-process Engine startups, which this PR did not
Qwen3.5 shard-index fix Main Not covered here; correct as is
refs/<revision> lookup This PR Same logic, but main copies it inline for cache_dir; this PR reuses find_local_repo_dir
Weight-listing fallback This PR (429 retry) Main's fallback lists the local snapshot, which at that point usually holds no weights, so every allowed weight format is downloaded
Offline check, --revision Main, with this PR's fix Main ignored --revision, so revision-pinned models (e.g. Inkling test) never went offline
Offline check, draft models Main, with this PR's fix Main's Engine hook required tokenizer files for drafts, which they usually lack, so in-process speculative startups never went offline

Changes

Area Change Scope
Launch checks GGUF, Mistral-format and DeepSeek FP8 checks read cached files before calling the Hub All
Single files LoRA configs, bitsandbytes, the token map and Inkling's quant config no longer list or snapshot the repo up front; only a Hub 404 or offline miss counts as a missing optional file All
429s Weight listing, snapshot downloads and diffusion Hub calls wait out a 429 All
Revisions Inkling's quant config, the processor's tokenizer reload and the tokenizer helpers use the requested revision; get_tokenizer / get_processor take one revision All
Eval data LongBench-v2 and MMMU load from commit-pinned cached files Tests
transformers lookups Revision lookups and repo listings served from the local cache (get_cached_repo_tree) CI
CI offline check Subprocess and Engine paths honor --revision and check drafts by config and weights only; cross-repo auto_map code is accepted CI

Verification

HF Hub api requests per launch (server + client), CI mode, warm cache. Outputs identical.

Launch main this PR
Qwen3-8B + KL helper (LongBench-v2) 38 0
Inkling --revision test, direct 67 0
Inkling --revision test, CI harness 67 (online) 0 (offline)
Llama-3.1-8B + LoRA 48 0

LongBench-v2 and all 30 MMMU subjects match the previous loads.


CI States

Latest PR Test (Base): ✅ Run #37894247607
Latest PR Test (Extra): ❌ Run #37894247194
Latest PR Test (AMD ROCm 10): ❌ Run #37894247523

@github-actions github-actions Bot added lora Multi-modal multi-modal language model diffusion SGLang Diffusion labels Oct 8, 2026
@mmangkad mmangkad added the highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed label Oct 8, 2026
@github-actions github-actions Bot added run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci) labels Oct 8, 2026
Mohammad Angkad and others added 2 commits October 9, 2026 06:15
…te-limit

# Conflicts:
#	python/sglang/srt/model_loader/weight_utils.py
#	python/sglang/srt/utils/common.py
#	python/sglang/test/test_utils.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

diffusion SGLang Diffusion highest-priority CI: all three control labels, plus never batch-cancelled or stale-closed lora Multi-modal multi-modal language model run-ci CI: run the baseline test suite on this PR run-ci-extra CI: also run the extra suite (requires run-ci)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant