Repository navigation
[Rust Frontend] Add /tokenize and /detokenize endpoints #44222
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
BugenZhao
merged 17 commits into
vllm-project:main
from
TanNgocDo:tando-feat/rust-tokenize-detokenize
Jun 9, 2026
Merged
Changes from 5 commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
e918a14
[Frontend][Rust] Add /tokenize and /detokenize endpoints to the Rust …
TanNgocDo 8747a89
Update rust/src/server/src/routes/openai/chat_completions/convert.rs
TanNgocDo 43a151b
Update rust/src/server/src/routes/openai/chat_completions/convert.rs
TanNgocDo de0f688
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo 728d29c
[Rust Frontend] Validate tokenize chat messages like chat completions
TanNgocDo 2e55d3e
Use openAI tool
TanNgocDo dee11ba
remove redundant token_id log
TanNgocDo 8242a24
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo 88ed1e8
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo d698787
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo fc45f45
[Rust Frontend] Address review nits for tokenize imports
TanNgocDo 4c6f1cd
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo 74589f7
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo 09710e3
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo 2e50de2
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo c3734e9
move `tokenize` / `detokenize` out of `openai` module
BugenZhao a94e8ca
Merge branch 'main' into tando-feat/rust-tokenize-detokenize
TanNgocDo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,10 @@ | ||
| pub mod chat_completions; | ||
| mod completions; | ||
| mod models; | ||
| mod tokenize; | ||
| pub(crate) mod utils; | ||
|
|
||
| pub use chat_completions::chat_completions; | ||
| pub use completions::completions; | ||
| pub use models::list_models; | ||
| pub use tokenize::{detokenize, tokenize}; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| //! `POST /tokenize` and `POST /detokenize` (root paths, matching Python). | ||
| //! | ||
| //! Encode/decode runs entirely in-process via [`DynTokenizer`]; the inference | ||
| //! engine is not involved. | ||
|
|
||
| mod types; | ||
|
|
||
| use std::sync::Arc; | ||
|
|
||
| use axum::Json; | ||
| use axum::extract::State; | ||
| use axum::http::HeaderMap; | ||
| use axum::response::{IntoResponse, Response}; | ||
| use thiserror_ext::AsReport as _; | ||
| use tracing::debug; | ||
|
|
||
| use crate::error::{ApiError, server_error}; | ||
| use crate::routes::openai::tokenize::types::{ | ||
| DetokenizeRequest, DetokenizeResponse, TokenizeChatRequest, TokenizeCompletionRequest, | ||
| TokenizeRequest, TokenizeResponse, | ||
| }; | ||
| use crate::routes::openai::utils::validated_json::ValidatedJson; | ||
| use crate::state::AppState; | ||
| use crate::utils::resolve_base_request_id; | ||
|
|
||
| /// Match Python `tokenize-{base}` where base is `X-Request-Id` or a new UUID. | ||
| fn tokenize_request_id(headers: &HeaderMap) -> String { | ||
| let base = resolve_base_request_id( | ||
| headers.get("X-Request-Id").and_then(|value| value.to_str().ok()), | ||
| None, | ||
| ); | ||
| format!("tokenize-{base}") | ||
| } | ||
|
|
||
| /// Reject an unknown model name, matching the other handlers. | ||
| fn check_model(state: &AppState, model: Option<&str>) -> Result<(), ApiError> { | ||
| if let Some(model) = model | ||
| && !state.served_model_names().iter().any(|n| n == model) | ||
| { | ||
| return Err(ApiError::model_not_found(model.to_string())); | ||
| } | ||
| Ok(()) | ||
| } | ||
|
|
||
| /// Build the `token_strs` vector when requested, via the tokenizer vocab. | ||
| fn token_strs(tokenizer: &vllm_text::tokenizer::DynTokenizer, ids: &[u32]) -> Vec<String> { | ||
| // Unknown IDs yield "" — intentional; matches Python's convert_ids_to_tokens behaviour. | ||
| ids.iter().map(|&id| tokenizer.id_to_token(id).unwrap_or_default()).collect() | ||
| } | ||
|
|
||
| pub async fn tokenize( | ||
| State(state): State<Arc<AppState>>, | ||
| headers: HeaderMap, | ||
| ValidatedJson(body): ValidatedJson<TokenizeRequest>, | ||
| ) -> Response { | ||
| let request_id = tokenize_request_id(&headers); | ||
| let tokenizer = state.chat.text().tokenizer(); | ||
| let max_model_len = state.chat.engine_core_client().max_model_len(); | ||
|
|
||
| let result = match body { | ||
| // Completion form: encode the raw `prompt` string (no chat template). | ||
| TokenizeRequest::Completion(req) => { | ||
| tokenize_completion(&state, &tokenizer, &request_id, req) | ||
| } | ||
| // Chat form: render `messages` through the template, then encode (see `tokenize_chat`). | ||
| TokenizeRequest::Chat(req) => tokenize_chat(&state, &request_id, req).await, | ||
| }; | ||
|
|
||
| match result { | ||
| Ok((tokens, want_strs)) => { | ||
| let token_strs = want_strs.then(|| token_strs(&tokenizer, &tokens)); | ||
| Json(TokenizeResponse { | ||
| count: tokens.len(), | ||
| max_model_len, | ||
| tokens, | ||
| token_strs, | ||
| }) | ||
| .into_response() | ||
| } | ||
| Err(error) => error.into_response(), | ||
| } | ||
| } | ||
|
|
||
| fn tokenize_completion( | ||
| state: &AppState, | ||
| tokenizer: &vllm_text::tokenizer::DynTokenizer, | ||
| request_id: &str, | ||
| req: TokenizeCompletionRequest, | ||
| ) -> Result<(Vec<u32>, bool), ApiError> { | ||
| debug!(%request_id, "tokenize completion"); | ||
|
TanNgocDo marked this conversation as resolved.
Resolved
|
||
| check_model(state, req.model.as_deref())?; | ||
| let tokens = tokenizer | ||
| .encode(&req.prompt, req.add_special_tokens) | ||
| .map_err(|e| server_error!("tokenize failed: {}", e.to_report_string()))?; | ||
| Ok((tokens, req.return_token_strs)) | ||
| } | ||
|
|
||
| /// HTTP adapter for the chat-shaped `/tokenize` body. | ||
| /// | ||
| /// Not [`vllm_chat::ChatLlm::tokenize_chat`]: this checks the model name and maps | ||
| /// errors to [`ApiError`]; the chat-crate method does render → finalize → encode. | ||
| async fn tokenize_chat( | ||
| state: &AppState, | ||
| request_id: &str, | ||
| req: TokenizeChatRequest, | ||
| ) -> Result<(Vec<u32>, bool), ApiError> { | ||
| debug!(%request_id, "tokenize chat"); | ||
| check_model(state, req.model.as_deref())?; | ||
| let return_token_strs = req.return_token_strs; | ||
| // `continue_final_message` / `add_generation_prompt` mutual exclusion is | ||
| // enforced in `normalize_generation_prompt_mode` inside `into_chat_request`. | ||
| let tokens = state | ||
| .chat | ||
| .tokenize_chat(req.into_chat_request(request_id.to_string())?) | ||
| .await | ||
| .map_err(|e| server_error!("tokenize failed: {}", e.to_report_string()))?; | ||
| Ok((tokens, return_token_strs)) | ||
| } | ||
|
|
||
| pub async fn detokenize( | ||
| State(state): State<Arc<AppState>>, | ||
| headers: HeaderMap, | ||
| ValidatedJson(body): ValidatedJson<DetokenizeRequest>, | ||
| ) -> Response { | ||
| let request_id = tokenize_request_id(&headers); | ||
| debug!(%request_id, "detokenize"); | ||
| if let Err(error) = check_model(&state, body.model.as_deref()) { | ||
| return error.into_response(); | ||
| } | ||
| let tokenizer = state.chat.text().tokenizer(); | ||
| match tokenizer.decode(&body.tokens, /* skip_special_tokens = */ false) { | ||
| Ok(prompt) => Json(DetokenizeResponse { prompt }).into_response(), | ||
| Err(e) => server_error!("detokenize failed: {}", e.to_report_string()).into_response(), | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use axum::http::{HeaderMap, HeaderValue}; | ||
|
|
||
| use super::tokenize_request_id; | ||
|
|
||
| #[test] | ||
| fn tokenize_request_id_prefers_x_request_id_header() { | ||
| let mut headers = HeaderMap::new(); | ||
| headers.insert("X-Request-Id", HeaderValue::from_static("client-req-1")); | ||
| assert_eq!(tokenize_request_id(&headers), "tokenize-client-req-1"); | ||
| } | ||
|
|
||
| #[test] | ||
| fn tokenize_request_id_generates_uuid_when_header_missing() { | ||
| let headers = HeaderMap::new(); | ||
| let id = tokenize_request_id(&headers); | ||
| assert!(id.starts_with("tokenize-")); | ||
| assert_ne!(id, "tokenize-"); | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.