Skip to content

[Misc] Refactor tokenizer interface#29693

Merged
vllm-bot merged 37 commits intovllm-project:mainfrom
DarkLight1337:tokenizer-proto
Nov 29, 2025
Merged

[Misc] Refactor tokenizer interface#29693
vllm-bot merged 37 commits intovllm-project:mainfrom
DarkLight1337:tokenizer-proto

Conversation

@DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Nov 28, 2025 •
edited by github-actions bot
Loading

Purpose

  • Consolidate AnyTokenizer and TokenizerBase by making TokenizerBase a Protocol, and rename both of them to TokenizerLike (with back-compatibility)
  • Remove unused attributes from TokenizerBase: sep_token, pad_token, encode_one
  • Move MistralTokenizer, TokenizerLike, TokenizerRegistry into vllm.tokenizers
  • Fix various type checking issues associated with replacing AnyTokenizer with TokenizerLike
  • Move tests from tests/tokenization to tests/tokenizers and update Buildkite pipeline accordingly
  • Consolidate related tests to fit the directory structure

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@DarkLight1337
[Misc] Convert TokenizerBase to protocol, consolidate tokenizer tests
bd9e0b3 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
BC
668eb2c 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@mergify
Copy link

mergify bot commented Nov 28, 2025

Documentation preview: https://vllm--29693.org.readthedocs.build/en/29693/

@mergify mergify bot added documentation Improvements or additions to documentation ci/build deepseek Related to DeepSeek models frontend llama Related to Llama models multi-modality Related to multi-modality (#4194) performance Performance-related issues qwen Related to Qwen models structured-output v1 labels Nov 28, 2025
@mergify mergify bot added the tool-calling label Nov 28, 2025
@DarkLight1337
Unnecessary quote
6684cd0 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Rename
9f3ab67 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Forward merge
d35e431 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Merge branch 'main' into tokenizer-proto
38afdfd 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Oops
a616c7a 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Docstring
94688c1 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Increase tolerance
54787a5 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
[Bugfix] Fix wrong mock attribute
bb46b1a 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Merge branch 'main' into tokenizer-proto
4ab6dcb 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Avoid circular import
c8e948d 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Merge branch 'fix-serving-test' into tokenizer-proto
665305e
@DarkLight1337
Fix mypy
94b9c62 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix circular import
3dbb92c 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
rel import
9aeed95 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@chatgpt-codex-connector
Copy link

chatgpt-codex-connector bot commented Nov 29, 2025

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

@DarkLight1337
Merge branch 'main' into tokenizer-proto
5137574
@DarkLight1337
Move
3866bae 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Nov 29, 2025

/gemini review

@DarkLight1337 DarkLight1337 changed the title [Misc] Convert TokenizerBase to protocol, consolidate tokenizer tests [Misc] Refactor tokenizer interface Nov 29, 2025
@DarkLight1337
Unnecessary runtime_checkable
c59476a 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
gemini-code-assist[bot]
gemini-code-assist bot reviewed Nov 29, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a significant and well-executed refactoring that modernizes the tokenizer abstraction by introducing the TokenizerLike protocol. This change improves modularity, type safety, and flexibility across the codebase. The consolidation of tokenizer tests into a dedicated tests/tokenizers directory is also a great organizational improvement.

I've identified a couple of critical bug fixes related to handling cases where the tokenizer is not initialized (skip_tokenizer_init=True), which significantly improves the robustness of the OpenAI-compatible endpoints. These changes prevent potential crashes and provide clearer error messages to the user.

Overall, this is an excellent contribution that enhances the maintainability and reliability of vLLM's tokenization system. The changes are thorough and consistent.

@DarkLight1337
Update import
9d974a3 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Avoid conflict with tokenizers package
15bf2a0 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Don't run type validation on internal structures
6b558be 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
kw only
767f2c8 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix mypy
02c1857 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Fix test
a5fbb67 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Nov 29, 2025

Failing tests are known failures on main

@vllm-bot vllm-bot merged commit 34a9842 into vllm-project:main Nov 29, 2025
129 of 134 checks passed
@DarkLight1337 DarkLight1337 deleted the tokenizer-proto branch November 29, 2025 12:02
kitaekatt pushed a commit to kitaekatt/vllm that referenced this pull request Dec 1, 2025
@DarkLight1337 @kitaekatt
[Misc] Refactor tokenizer interface (vllm-project#29693)
97546bc 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
amd-hhashemi pushed a commit to amd-hhashemi/vllm that referenced this pull request Dec 2, 2025
@DarkLight1337 @amd-hhashemi
[Misc] Refactor tokenizer interface (vllm-project#29693)
d8d85bb 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Hashem Hashemi <hashem.hashemi@amd.com>
@Rohan138 Rohan138 mentioned this pull request Dec 4, 2025
Merged
5 tasks
dsuhinin pushed a commit to dsuhinin/vllm that referenced this pull request Jan 21, 2026
@DarkLight1337 @dsuhinin
[Misc] Refactor tokenizer interface (vllm-project#29693)
0b2d55a 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: dsuhinin <suhinin.dmitriy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Isotr0py Isotr0py Isotr0py approved these changes

@patrickvonplaten patrickvonplaten Awaiting requested review from patrickvonplaten patrickvonplaten is a code owner

@sighingnow sighingnow Awaiting requested review from sighingnow sighingnow is a code owner

@hmellor hmellor Awaiting requested review from hmellor hmellor is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm is a code owner

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@russellb russellb Awaiting requested review from russellb russellb is a code owner

@benchislett benchislett Awaiting requested review from benchislett benchislett is a code owner

@chaunceyjiang chaunceyjiang Awaiting requested review from chaunceyjiang chaunceyjiang is a code owner

@tjtanaa tjtanaa Awaiting requested review from tjtanaa tjtanaa is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build deepseek Related to DeepSeek models documentation Improvements or additions to documentation frontend llama Related to Llama models multi-modality Related to multi-modality (#4194) performance Performance-related issues qwen Related to Qwen models ready ONLY add when PR is ready to merge/full CI is needed ready-run-all-tests Trigger CI with all tests for wide-ranging PRs structured-output tool-calling v1

Projects

Structured Output
Status: Done
Tool Calling
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@DarkLight1337 @Isotr0py @vllm-bot