Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion model_gateway/src/core/steps/tokenizer_registration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ impl StepExecutor<TokenizerWorkflowData> for LoadTokenizerStep {
// This handles: validation, deduplication, and loading
let result = app_context
.tokenizer_registry
.load(&id, &name, &source, || {
.load(&id, &name, &source, chat_template.as_deref(), || {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

Remove caller-supplied chat_template_path from registry metadata input.

At Line 108, chat_template.as_deref() is passed into tokenizer_registry.load(...). This makes registry metadata caller-driven and can drift from the actually loaded tokenizer state (notably on remote-fallback paths). Keep load() deriving metadata from the tokenizer instance/trait and avoid passing this field at the call site.

♻️ Suggested call-site direction
-            .load(&id, &name, &source, chat_template.as_deref(), || {
+            .load(&id, &name, &source, || {
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@model_gateway/src/core/steps/tokenizer_registration.rs` at line 108,
Call-site is passing caller-supplied chat_template (chat_template.as_deref())
into tokenizer_registry.load, which lets callers override registry metadata;
remove that argument and let load() derive chat_template_path from the loaded
tokenizer instance. Specifically, in tokenizer_registration.rs replace the
.load(&id, &name, &source, chat_template.as_deref(), || { ... }) call with
.load(&id, &name, &source, || { ... }) (i.e., drop the
chat_template/chat_template_path argument) and ensure any code expecting that
parameter is adjusted so metadata is computed inside tokenizer_registry.load or
the provided factory closure using the tokenizer trait methods instead of from
the caller.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve chat template metadata only when it is actually applied

Passing chat_template.as_deref() into tokenizer_registry.load unconditionally records the requested template path even when local creation fails and the flow falls back to fetch_tokenizer_from_worker, which builds the tokenizer with None for chat templates. In that fallback scenario (common in IGW mode when the gateway cannot load locally), /v1/tokenizers and /v1/tokenizers/{id} will report a chat_template_path that was never used, so the new metadata is inaccurate.

Useful? React with 👍 / 👎.

let source = source.clone();
let chat_template = chat_template.clone();
let cache_cfg = cache_config.clone();
Expand Down
10 changes: 8 additions & 2 deletions model_gateway/src/routers/tokenize/handlers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,10 @@ pub async fn list_tokenizers(registry: &Arc<TokenizerRegistry>) -> Response {
id: e.id,
name: e.name,
source: e.source,
vocab_size: e.tokenizer.vocab_size(),
vocab_size: e.vocab_size, // Use pre-calculated size from entry
chat_template_path: e.chat_template_path,
Comment thread
ppraneth marked this conversation as resolved.
Resolved
backend: e.backend.to_string(),
created_at: e.created_at,
})
.collect();

Expand Down Expand Up @@ -338,7 +341,10 @@ pub async fn get_tokenizer_info(context: &Arc<AppContext>, tokenizer_id: &str) -
id: e.id,
name: e.name,
source: e.source,
vocab_size: e.tokenizer.vocab_size(),
vocab_size: e.vocab_size,
chat_template_path: e.chat_template_path,
backend: e.backend.to_string(),
created_at: e.created_at,
};
Json(info).into_response()
}
Expand Down
4 changes: 4 additions & 0 deletions multimodal/src/registry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,10 @@ mod tests {
fn as_any(&self) -> &dyn std::any::Any {
self
}

fn backend(&self) -> llm_tokenizer::TokenizerBackend {
llm_tokenizer::TokenizerBackend::Mock
}
}

#[test]
Expand Down
2 changes: 1 addition & 1 deletion protocols/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ axum = ["dep:axum"]
[dependencies]
axum = { workspace = true, optional = true }
bitflags.workspace = true
chrono.workspace = true
chrono = { workspace = true, features = ["serde"] }
rand.workspace = true
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["preserve_order"] }
Expand Down
4 changes: 4 additions & 0 deletions protocols/src/tokenize.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
//! These types mirror the SGLang Python implementation for compatibility.
//! See: python/sglang/srt/entrypoints/openai/protocol.py

use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};

use super::UNKNOWN_MODEL_ID;
Expand Down Expand Up @@ -161,6 +162,9 @@ pub struct TokenizerInfo {
/// Source path or HuggingFace model ID
pub source: String,
pub vocab_size: usize,
pub chat_template_path: Option<String>,
pub backend: String,
pub created_at: DateTime<Utc>,
}

/// Request schema for removing a tokenizer
Expand Down
3 changes: 2 additions & 1 deletion tokenizer/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ name = "llm_tokenizer"
[dependencies]
anyhow.workspace = true
blake3.workspace = true
chrono.workspace = true
bytemuck = { workspace = true, features = ["derive"] }
dashmap.workspace = true
lru.workspace = true
Expand Down Expand Up @@ -51,4 +52,4 @@ name = "stop_sequence_search"
harness = false

[lints]
workspace = true
workspace = true
4 changes: 4 additions & 0 deletions tokenizer/src/cache/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,10 @@ impl Tokenizer for CachedTokenizer {
self
}

fn backend(&self) -> crate::traits::TokenizerBackend {
self.inner.backend()
}

fn apply_chat_template(
&self,
messages: &[serde_json::Value],
Expand Down
4 changes: 4 additions & 0 deletions tokenizer/src/huggingface.rs
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,10 @@ impl TokenizerTrait for HuggingFaceTokenizer {
self
}

fn backend(&self) -> crate::traits::TokenizerBackend {
crate::traits::TokenizerBackend::HuggingFace
}

fn apply_chat_template(
&self,
messages: &[serde_json::Value],
Expand Down
1 change: 1 addition & 0 deletions tokenizer/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ pub use stream::DecodeStream;
pub use tiktoken::{TiktokenModel, TiktokenTokenizer};
pub use traits::{
Decoder, Encoder, Encoding, SpecialTokens, TokenIdType, Tokenizer as TokenizerTrait,
TokenizerBackend,
};

/// Main tokenizer wrapper that provides a unified interface for different tokenizer implementations
Expand Down
4 changes: 4 additions & 0 deletions tokenizer/src/mock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -125,4 +125,8 @@ impl TokenizerTrait for MockTokenizer {
fn as_any(&self) -> &dyn std::any::Any {
self
}

fn backend(&self) -> crate::traits::TokenizerBackend {
crate::traits::TokenizerBackend::Mock
}
}
Loading