[dskwe]

Overview

Honor ModelConfig.code_revision in both config-based tokenizer helpers when the model and tokenizer share a repository. Complements #61272, which fixed snapshot resolution for direct tokenizer loading.

Claims

• Config-based loading no longer silently ignores an independently pinned tokenizer code revision.
• A separately configured tokenizer does not inherit an unrelated model commit.
• Explicit caller overrides, including code_revision=None, remain authoritative.

Validation

Config-code pin follow-up (4a34bfa32d)

get_tokenizer now forwards the resolved code_revision to get_config, so the remote AutoConfig module follows the same pin as tokenizer code. This is an appended commit; earlier reviewed commits are unchanged.

• New regression uses real HF dynamic config loading from hf-internal-testing/test_dynamic_model, with only tokenizer instantiation replaced. Default and matching revisions pass before the fix; a distinct pin fails because the loaded config class comes from the data revision. Negative control: 1 failed, 2 passed. With the fix, the full code-revision selection is 26 passed, 5 deselected.
• Registry, generate-stream and LoRA resolver suites: 109 passed in the existing native Linux CPU validation container.
• All applicable pinned pre-commit hooks pass, including mypy and Buildkite test tethering.
• A separate real-backend smoke probe (no tokenizer-loader replacement) used local tokenizer assets with a remote AutoConfig reference. Baseline loaded config code from c8ae8a2e5845fc3ad4e5237e4346e5a9ba1e87d2; the fix loaded the requested 2efddce40dddaccd37bae208c3c7ca66dbedf68a. Both encoded hello world to [1, 2]. This is tokenizer validation, not model inference or a new HTTP serving test.

Review follow-up (554c1c1842)

Both review findings are addressed: resolved Mistral mode drops the HF-only code_revision key; both config helpers tolerate a missing attribute and no longer inject unset revisions. HF-backed aliases and explicit caller overrides remain supported.

Native pytest was rerun in the existing Linux CPU container (Python 3.12.13, Torch 2.13.0+cpu, Transformers 5.19.0, pytest 9.1.1), with this worktree's Python package and tests copied into an isolated /validation/pr61611 directory. No AST extraction was used for these results.

Check	Result
python -m pytest tests/tokenizers_/test_registry.py -q --tb=short	57 passed; original PR registry with the new tests: 20 failed, 37 passed
python -m pytest tests/tokenizers_/test_basic.py -k code_revision -q --tb=short	23 passed, 5 deselected (includes 8 real Mistral backend cases)
python -m pytest tests/tokenizers_/test_basic.py -k config_mistral_code_revision -q --tb=no against the original PR registry	All 8 failed; real backend rejects code_revision
python -m pytest tests/entrypoints/scale_out/token_in_token_out/test_generate_stream.py tests/entrypoints/openai/completion/test_lora_resolvers.py -q --tb=short	52 passed, without changing their minimal config stubs
pre-commit run --files vllm/tokenizers/registry.py tests/tokenizers_/test_registry.py tests/tokenizers_/test_basic.py	All applicable hooks passed, including mypy and Buildkite test tethering

The Mistral tests load mistralai/Mistral-7B-Instruct-v0.3 through both helpers in explicit and auto modes, with code_revision=None and "never". Original registry behavior was tested in the same environment with automatic restoration of the fixed file. Warnings were Torch JIT deprecations and the existing TestTokenizer collection warning. These are real tokenizer/backend tests, not model-weight loading, inference/GPU evaluation, or a new HTTP serving test.

Initial validation

Linux x86_64, Python 3.12.13, Torch 2.13.0+cpu, Transformers 5.19.0. Validation reused a native environment based on e0317b2 with the relevant current modules and tests copied in; it was not a complete native rebuild of b9b401b.

Check	Result
Config/code-revision regressions against baseline	4 failed, 11 passed, 5 deselected: both helpers silently accepted unknown or missing-code revisions
Same tests with this fix	15 passed, 5 deselected
tests/tokenizers_/test_registry.py	7 passed
Applicable pre-commit hooks on all three changed files	Passed, including mypy and Buildkite test tethering
git diff --check	Passed

A real ModelConfig → VllmConfig → renderer integration also verified separate-repository loading and explicit None overrides.

A real no-weights vllm launch render HTTP reproducer confirmed:

• Baseline: unknown and missing-code revisions incorrectly allow startup and HTTP 200 from /tokenize.
• Fixed: both are rejected during tokenizer initialization with the corresponding Transformers OSError.
• Fixed matching revision: HTTP 200 with token IDs [1, 0, 0, 2].

How to Test

.venv/bin/python -m pytest tests/tokenizers_/test_basic.py -k code_revision -v --tb=short
.venv/bin/python -m pytest tests/tokenizers_/test_registry.py -v --tb=short
pre-commit run --files vllm/tokenizers/registry.py tests/tokenizers_/test_basic.py tests/tokenizers_/test_registry.py

For manual reproduction, use hf-internal-testing/test_dynamic_tokenizer as both model and tokenizer, pinned to 08a35780b7a78794436d8654aafa26ee0165c149, with --tokenizer-mode hf --trust-remote-code --max-model-len 32. The tokenizer-only fixture needs --hf-config-path pointing to a real cached OPT config (tested with facebook/opt-125m at 27dcfa74d334bc871f3234de431e71c6eeba5dd6). Keep Hub access enabled. Set --code-revision never or 73293ac504e43f4a895688b57c65cebe24f01a38: baseline starts, but this fix rejects the requested code. A matching code revision starts successfully and /tokenize returns the control token IDs above.

Details

Both tokenizer_args_from_config and cached_tokenizer_from_config omitted the model's code revision. They now use setdefault and the same model/tokenizer repository equality guard already used by the processor chat-template path. This avoids applying a model commit to an independently configured tokenizer repository. The skip-tokenizer-init early return is unchanged.

• vllm/tokenizers/registry.py: guarded code-revision propagation in both wrappers.
• tests/tokenizers_/test_basic.py: real remote-code loading regressions for both helpers, separate repositories and explicit overrides; real Mistral config-loading regressions; accept equivalent offline errors in the existing negative test.
• tests/tokenizers_/test_registry.py: complete the existing config fixture and regress Mistral argument filtering, minimal configs, HF-backed aliases and explicit overrides.

Duplicate searches were refreshed October 14, 2026 for code_revision tokenizer, both helper names, and #61272 references. No matching open wrapper fix was found. #61210 concerns speculative MTP config; #51446 and #42616 fix other artifact boundaries; #61272 fixes the downstream resolver, not these wrappers.

AI assistance was used for investigation, implementation, validation and review. No inference accuracy or performance improvement is claimed. Full tokenizer-suite, model-inference/GPU evaluations, repository-wide lint and a full current-HEAD native rebuild were not run; the HTTP evidence is render-only tokenization, not generation.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

@claude review

[DarkLight1337]

@claude review

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

• 🟣 vllm/tokenizers/registry.py — Pre-existing, security-relevant: operators pinning --code-revision still run the tokenizer config's remote code from --revision, even though this PR now forwards code_revision. get_tokenizer calls get_config with trust_remote_code and revision but no code_revision (registry.py:247-253). It then reuses that config through kwargs.setdefault("config", config). Fix: pass code_revision=kwargs.get("code_revision") to this get_config call, so config code and tokenizer code come from the same pinned commit. The Mistral path already pops the key before this point.

  Why this was flagged

  Trigger: a trust_remote_code model whose config.json has an auto_map AutoConfig entry, served with --revision R --code-revision C where R != C. Model and tokenizer share a repo. cached_tokenizer_from_config now adds code_revision=C to kwargs (registry.py:298-301). Because C differs from R, resolve_tokenizer_args skips the snapshot path (registry.py:168) and keeps the repo id. get_tokenizer then calls get_config at registry.py:247-253 without code_revision. get_config (transformers_utils/config.py:851-855) accepts code_revision, but here it defaults to None, so Transformers imports and runs configuration_*.py from R. That config object is handed to AutoTokenizer at registry.py:271. The base branch also ran R's config code here, so this is pre-existing. The PR's stated goal of honoring code_revision in tokenizer loading is therefore only partly met.

  Verification: Pre-existing. get_tokenizer calls get_config without code_revision (registry.py:247-253), though get_config accepts it (transformers_utils/config.py:855). When C != R, Transformers falls back to revision, so the remote configuration module runs from R and is reused via kwargs.setdefault("config", config) (registry.py:271).

[DarkLight1337]

Thanks

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #95320 for commit 1ae0564be03f.