[richard-for-bugen]

Summary

Python's --skip-tokenizer-init serves token IDs only: no tokenizer is loaded, prompts must be token IDs, and responses carry token IDs with empty text. The Rust frontend declares the flag Unsupported, so a server started with it exits at startup. Under vllm serve with VLLM_USE_RUST_FRONTEND=1, the flag reaches Rust through --args-json and fails the same way, although tests/entrypoints/openai/completion/test_token_in_token_out.py (a model directory without tokenizer files) is exactly this use case.

This PR implements the flag. Behavior with it:

Input	Python	Rust after this PR
/v1/completions with a token-ID prompt	runs; text is "", token IDs returned on request	same
/inference/v1/generate, gRPC generate	runs	runs (token-in/token-out already)
text prompt, chat request, /tokenize, /detokenize	error (Tokenizer not available when skip_tokenizer_init=True)	400 "this request needs a tokenizer, which is not loaded when `--skip-tokenizer-init` is set"
echo of a token-ID prompt without return_token_ids	AssertionError (500)	400
structured outputs	400	400 on inference (the engine skips its tokenizer too, so it cannot compile grammars); the render server forwards them, since it does not configure the downstream engine
stop strings, bad_words	silently ignored (no detokenizer, update_from_tokenizer skipped)	400: neither can take effect without a tokenizer
completions logprobs	error when a token string is needed; return_tokens_as_token_ids works	400 unless return_tokens_as_token_ids (request or server default) applies; choice-level prompt_logprobs (keyed by token ID) and /inference/v1/generate logprobs work
token-ID range check	model vocab size (vllm-project#62911)	model vocab size
primary EOS	None; generation_config EOS ids become stop tokens	first config.json EOS id (Rust's existing fallback when the tokenizer has no EOS); the rest stay stop tokens

Design and layering

The tokenizer is a null object, not an Option. Under the flag the text backend holds an UnavailableTokenizer whose text operations fail with a dedicated error, so the Tokenizer interface and its consumers (prompt encoding, tokenize_bad_words, /tokenize, /detokenize, echo decoding) are unchanged and fail through their existing error paths. Only two places know the tokenizer is absent: loading, and one TextBackend::has_tokenizer flag for the checks that have no tokenizer call to fail.

• Tokenizer (vllm-tokenizer). TokenizerError becomes an enum: Backend { message } for the existing failures, and Unavailable. UnavailableTokenizer returns Unavailable from encode, encode_ordinary and decode, has no token strings, and reports the model vocabulary as its size, which bounds token-ID prompts like Python ([Bugfix][Core] Validate prompt token ids when skip_tokenizer_init is set vllm-project/vllm#62911). The rest of this crate's diff is the mechanical rename of the error macro (tokenizer_error! to backend!).
• Errors (vllm-text, vllm-chat). TokenizerError::Unavailable maps to vllm_text::Error::TokenizerUnavailable, a request validation error, so every rejection is a 400 with one message. /tokenize and /detokenize now map tokenizer errors through text_submit_error instead of a blanket 500, so other tokenizer failures keep their 500 and only Unavailable becomes a 400. Side effect: other request validation errors from /tokenize (for example prompt truncation) become 400 too, instead of 500.
• CLI (vllm-cmd). skip_tokenizer_init moves from the Unsupported markers to ModelArgs, next to --tokenizer, --revision and --hf-config-path, because like them it decides which model files are read. It flows into ModelFileSources, and ModelArgs::python_args forwards it to the managed engine under vllm-rs serve: the engine's structured-output manager would otherwise load a tokenizer, which fails when the files are absent. The render server shares ModelArgs.
• Loading (vllm-text, vllm-chat). ModelFileSources::skip_tokenizer_init reads no tokenizer-side file (tokenizer, tokenizer_config.json, chat template; an explicit --tokenizer source is not opened), so a cached snapshot only needs config.json. HfTextBackend builds the UnavailableTokenizer when no tokenizer source is resolved, and load_model_backends builds a TokenizerlessChatBackend instead of HfChatBackend: rendering and output parsing both need a tokenizer, so it rejects every chat request.
• The flag. TextRequestProcessor::prepare sets decode_options.detokenize = false when there is no tokenizer, which reuses the token-only output path (TokenOnlyDecoder) and the stop-string rejection of [Perf][Rust Frontend] Skip detokenization for gRPC requests that read no text vllm-project/vllm#63789. Two more checks read it, because their consumer is not a tokenizer call in this process: structured outputs are rejected in TextLlm just before engine submission, since the engine skips its tokenizer too (the render server, which shares prepare, forwards them to an engine it does not configure); completions logprobs are rejected up front unless return_tokens_as_token_ids applies, because id_to_token returning None would key every candidate by the same empty string instead of failing.

Validation

• cargo fmt --check, cargo sort --workspace --check, cargo deny check bans, cargo clippy --workspace --all-targets --all-features --locked -D warnings: clean.
• cargo nextest run --workspace --all-features --locked: 2408 passed, 13 skipped.
• New tests: vllm-text tokenizerless_backend_prepares_token_id_prompts_only (token-ID prompt lowers with detokenization off; text prompt, stop strings, bad_words rejected; structured outputs survive prepare for the render server; model-vocab bound) and skip_tokenizer_init_loads_a_model_without_tokenizer_files (resolution and backend load ignore a missing --tokenizer source and an unparsable tokenizer_config.json; EOS from the model config); vllm-chat skip_tokenizer_init_rejects_every_chat_request; vllm-server skip_tokenizer_init_serves_token_id_completions_only (token-ID completion with return_token_ids returns text: "" and the token IDs; text prompt, echo, stop, string-form logprobs, structured outputs, /tokenize, /detokenize are 400); vllm-cmd skip_tokenizer_init_reaches_backends_and_managed_engine (CLI and --args-json, forwarded to the managed engine).
• E2E on one GB200 GPU, tests/entrypoints/openai/completion/test_token_in_token_out.py (Qwen3-0.6B with the tokenizer files removed, --skip-tokenizer-init, dummy weights): with this branch's vllm-rs 1 passed; with main's vllm-rs 1 error (the frontend rejects skip_tokenizer_init in --args-json at startup); Python frontend 1 passed.

Notes

• The stop-string and bad_words rejections are deliberate divergences: Python accepts them and silently does nothing, which the Rust frontend's "reject what is not implemented" rule forbids.
• gpt-oss chat is rejected too. Python's Harmony path happens to render and parse without an HF tokenizer, but nothing tests or documents chat under --skip-tokenizer-init (Python only tests that the chat server object starts without one), so this PR rejects every chat request uniformly. Keeping a tokenizer-optional Harmony chat backend is a possible follow-up.
• test_token_in_token_out.py can join the Rust frontend CI step once this merges.

Codex review

This revision replaces the Option<DynTokenizer> design (old head 4112478) with the null tokenizer. Codex (gpt-6.1-sol, high) on the new design: changes needed, with the same two P2 findings as on the old design, both rejected for the same reasons, and one nit left open.

• Rejected, P2: "keep native Harmony chat and chat /tokenize without the HF tokenizer". Python's Harmony path does work without one, but only incidentally; nothing tests or documents chat under --skip-tokenizer-init. Supporting it means making HfChatBackend tokenizer-optional, a follow-up (see Notes).
• Rejected, P2: "an external engine may have a tokenizer, so forward structured outputs". skip_tokenizer_init is one ModelConfig field shared by frontend and engine: Python's launcher and vllm-rs serve pass it to both, and Python's frontend rejects structured outputs under it regardless of the engine.
• Open, P3: the successful exemptions (logprobs with return_tokens_as_token_ids, streaming) have no route test. Covering them needs a mock engine that emits logprobs; the condition itself is one boolean next to the rejection that is tested.

It confirmed the rest of the layering (file loading in ModelArgs → ModelFileSources → HfTextBackend, token-only output through TokenOnlyDecoder, the logprobs guard protecting decoded_logprobs_to_openai) and found no other correctness issue.

[richard-for-bugen]

Redesigned as you suggested, force-pushed over the old head 4112478 (now ec3f71c, rebased on the synced fork main 53624fc).

• The tokenizer is a null object, UnavailableTokenizer, instead of Option<DynTokenizer>. Its text operations fail with a dedicated TokenizerError::Unavailable, which maps to vllm_text::Error::TokenizerUnavailable (400). The Tokenizer interface and its consumers are unchanged.
• Only loading and one TextBackend::has_tokenizer flag know the tokenizer is absent. prepare uses the flag to force detokenize = false, reusing the TokenOnlyDecoder path and stop-string rejection from [Perf][Rust Frontend] Skip detokenization for gRPC requests that read no text vllm-project/vllm#63789; the structured-output and string-logprobs checks also read it.
• Size: production code +485/−78 against +519/−141 before. About 50 lines of that are the mechanical macro rename caused by TokenizerError becoming an enum.

CI command set green (2408 passed, 13 skipped); GB200 e2e test_token_in_token_out.py passes. Codex repeated its two earlier P2s (Harmony chat without a tokenizer, external engine with a tokenizer); both are rejected for the same reasons as before, see the body.

[BugenZhao]

Suggested change

	if !self.backend.has_tokenizer() {
	if !self.has_tokenizer() {

[richard-for-bugen]

Applied in 47f09f8.

[richard-for-bugen]

Promoted as vllm-project#64697 (one squashed commit e3008f8 on main 8559100, tree identical to the staging head 46faac6). This PR stays open until it merges.

[richard-for-bugen]

Merged upstream as vllm-project#64697; closing.