[vllm + v5 fix] handle TokenizersBackend fallback properly for v5#44255

Merged

ArthurZucker merged 48 commits intomainfrom

bad_models_update

Mar 4, 2026

Commits on Mar 4, 2026

update deepseek v2 for tokenizers v5

ita.zaporozhets@huggingface.co
authored and
itazap
committed
adding remote code fix

ita.zaporozhets@huggingface.co
authored and
itazap
committed
fix deepseek name

ita.zaporozhets@huggingface.co
authored and
itazap
committed
handle spm conversion from proto only when overriding bad_models
itazap
committed
add script to compare xlni and code_search_net output of 2 tokenizers
itazap
committed
tiktoken models support
itazap
committed
fix tests
itazap
committed
testssss
itazap
committed
fix gemma
itazap
committed
apply some feedback
itazap
committed
paligemma processor tests fix
itazap
committed
add relevant changes from #44298
itazap
committed
json serializable fix
itazap
committed
add more xlni cases
itazap
committed
t5 fix
itazap
committed
ruff check code quality
itazap
committed
missed file for t5 test fix
itazap
committed
modular failures
itazap
committed
other modular fixes
itazap
committed
tiktoken.model test
itazap
committed
more feedback updates!
itazap
committed
fixing models so AutoTokenizer == TokenizersBackend - aligning with converters
itazap
committed
seamless m4t
itazap
committed
missed the most important files
itazap
committed
Revert "missed the most important files"
itazap
committed
undo changes to big bird , bert, seamless
itazap
committed
setup and qual
itazap
committed
lasr
itazap
committed
t5
itazap
committed
dpr bert
itazap
committed
xlmroberta
itazap
committed
reformer
itazap
committed
nllb
itazap
committed
style and shit
ArthurZucker
committed
update
ArthurZucker
committed
fix
ArthurZucker
committed
extract the charsmap
ArthurZucker
committed
fix mbart?
ArthurZucker
committed
style
ArthurZucker
committed
nllb and test tok common read spm precompiled charsmap
itazap
committed
fix whisper?
ArthurZucker
committed
Merge branch 'bad_models_update' of github.com:huggingface/transformers into bad_models_update
ArthurZucker
committed
nllb
itazap
committed
checked on v4!
ArthurZucker
committed
Merge branch 'bad_models_update' of github.com:huggingface/transformers into bad_models_update
ArthurZucker
committed
fix repo
ArthurZucker
committed
fix lasr
ArthurZucker
committed
style
ArthurZucker
committed