[ppraneth]

Problem

Closes #28

Solution

This PR enriches the TokenizerEntry structure and the corresponding TokenizerInfo API response with comprehensive metadata.

• Backend Identification: Introduced a TokenizerBackend enum and a backend() method on the Tokenizer trait.
• Automatic Population: Updated TokenizerRegistry::load to automatically capture:
  • vocab_size
  • detected backend
  • created_at timestamp
• API Exposure: Updated the TokenizerInfo protocol to expose these new fields to API clients.

---

Changes

tokenizer crate

• Added TokenizerBackend enum in traits.rs.

• Extended TokenizerEntry with:

  • vocab_size
  • chat_template_path
  • backend
  • created_at

• Implemented backend() for:

  • HuggingFaceTokenizer
  • TiktokenTokenizer
  • MockTokenizer
  • CachedTokenizer

• Updated TokenizerRegistry::load to populate metadata automatically.

• Re-exported TokenizerBackend in lib.rs.

protocols crate

• Updated TokenizerInfo to include the new metadata fields.
• Enabled the serde feature for the chrono dependency.

model_gateway crate

• Updated LoadTokenizerStep to pass chat template paths correctly.
• Updated handlers to map TokenizerEntry metadata to TokenizerInfo.

multimodal crate

• Updated TestTokenizer to implement the new backend() trait method.

---

Test Plan

Added new unit tests in tokenizer/src/registry.rs to verify metadata population and registry behavior.

1. Verify Metadata Population

cargo test --package llm-tokenizer --lib registry::tests::test_metadata_population

Asserts that vocab_size is correctly captured, backend is identified as Mock, and timestamps are valid.

2. Verify Registry Immutability

cargo test --package llm-tokenizer --lib registry::tests::test_load_ignores_updates

Asserts that attempting to reload a tokenizer with different parameters does not corrupt the existing entry.

3. General Regression

Ran full test suite for tokenizer crate:

cargo test --package llm-tokenizer

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes
• (Optional) Documentation updated

Summary by CodeRabbit

• New Features
  • Tokenizer endpoints return richer metadata: chat template path, backend type (HuggingFace, Tiktoken, Mock), and creation timestamp.
  • Tokenizers expose their backend type at runtime.
  • Tokenizer loading accepts an optional chat-template context for deduplication/association.
  • Vocabulary size is now pre-calculated and returned for improved performance.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds tokenizer metadata (vocab_size, chat_template_path, backend, created_at), exposes TokenizerBackend and backend() on tokenizers, updates registry load/register to accept chat_template_path, and surfaces metadata in API TokenizerInfo and related changes across crates.

Changes

Cohort / File(s)	Summary
Tokenizer trait & export tokenizer/src/traits.rs, tokenizer/src/lib.rs	Adds TokenizerBackend enum, backend(&self) -> TokenizerBackend to the Tokenizer trait, implements Display, and re-exports TokenizerBackend.
Tokenizer implementations tokenizer/src/huggingface.rs, tokenizer/src/tiktoken.rs, tokenizer/src/mock.rs, tokenizer/src/cache/mod.rs	Implements backend() for HuggingFace, Tiktoken, Mock; CachedTokenizer delegates to inner tokenizer.
Registry & entry metadata tokenizer/src/registry.rs	Adds vocab_size, chat_template_path, backend, created_at to TokenizerEntry; adds TokenizerEntry::new; updates load and register signatures to accept chat_template_path; updates internal construction and tests.
API shapes & handlers protocols/src/tokenize.rs, model_gateway/src/routers/tokenize/handlers.rs	Adds chat_template_path: Option<String>, backend: String, created_at: DateTime<Utc> to TokenizerInfo; handlers now source vocab_size from entry and include new fields in responses.
Registration callsite model_gateway/src/core/steps/tokenizer_registration.rs	Passes chat_template.as_deref() into TokenizerRegistry::load() call.
Test utilities & mocks multimodal/src/registry.rs, tokenizer/src/mock.rs	Test/mock tokenizer implements backend(); tests adjusted to provide chat_template_path (often None).
Dependency adjustments protocols/Cargo.toml, tokenizer/Cargo.toml	Enables chrono with serde feature in protocols; adds workspace chrono to tokenizer.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat(gRPC): Add GetTokenizer Proto and Tokenizer Bundle Streaming #470 — Modifies tokenizer registration load site to pass an optional chat template into registry load (closely related change).
• feat(tokenizer): add tiktoken hub model support and unify chat template API #439 — Also extends TokenizerRegistry::load to accept chat_template path and propagate it through loading (strongly related).

Suggested labels

tests

Suggested reviewers

• CatherineSue
• key4ng
• slin1237

Poem

🐰 A rabbit found tokens, neat and spry,

Backends named and timestamps high,
Chat templates tucked in a cozy nest,
Vocab sizes logged, each one blessed,
Hoppity hops — the registry's dressed!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Linked Issues check	❓ Inconclusive	PR implements core requirements (TokenizerBackend enum, metadata fields, load/register updates, API responses) but reviewer feedback indicates a design issue with chat_template_path handling that may represent incomplete fulfillment of issue intent.	Address reviewer concern about chat_template_path derivation: remove it from load() parameters and implement it as a trait method on Tokenizer instead, or derive it from tokenizer storage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: enriching tokenizer metadata with backend, vocab_size, and creation time.
Out of Scope Changes check	✅ Passed	All changes directly support the PR objective of enriching tokenizer metadata; no unrelated modifications detected across affected files.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch ppraneth/meta

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @ppraneth, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the tokenizer system by integrating comprehensive metadata directly into TokenizerEntry and exposing it via the TokenizerInfo API. This provides richer information about each tokenizer, including its backend type, vocabulary size, chat template path, and creation timestamp, improving observability and management of tokenizers within the system.

Highlights

• Enhanced Tokenizer Metadata: The TokenizerEntry structure has been enriched with new metadata fields: vocab_size, chat_template_path, backend, and created_at.
• Tokenizer Backend Identification: A new TokenizerBackend enum and a backend() method have been introduced to the Tokenizer trait, allowing for explicit identification of the tokenizer's underlying implementation (e.g., HuggingFace, Tiktoken, Mock).
• Automated Metadata Population: The TokenizerRegistry::load and register methods now automatically capture and populate the new metadata fields when a tokenizer is loaded or registered.
• API Exposure of Metadata: The TokenizerInfo API response has been updated to expose all the newly added tokenizer metadata fields to API clients.

Changelog

• model_gateway/src/core/steps/tokenizer_registration.rs
  • Updated the tokenizer_registry.load call to pass the chat_template path.
• model_gateway/src/routers/tokenize/handlers.rs
  • Modified list_tokenizers and get_tokenizer_info to map new metadata fields from TokenizerEntry to TokenizerInfo.
• multimodal/src/registry.rs
  • Implemented the new backend() trait method for TestTokenizer.
• protocols/Cargo.toml
  • Enabled the serde feature for the chrono dependency.
• protocols/src/tokenize.rs
  • Imported DateTime and Utc from chrono.
  • Added chat_template_path, backend, and created_at fields to the TokenizerInfo struct.
• tokenizer/Cargo.toml
  • Added chrono as a dependency.
• tokenizer/src/cache/mod.rs
  • Implemented the backend() trait method for CachedTokenizer.
• tokenizer/src/huggingface.rs
  • Implemented the backend() trait method for HuggingFaceTokenizer.
• tokenizer/src/lib.rs
  • Re-exported TokenizerBackend from the traits module.
• tokenizer/src/mock.rs
  • Implemented the backend() trait method for MockTokenizer.
• tokenizer/src/registry.rs
  • Imported chrono types and TokenizerBackend.
  • Extended TokenizerEntry with vocab_size, chat_template_path, backend, and created_at fields.
  • Added a new constructor for TokenizerEntry to simplify initialization.
  • Updated load and register methods to accept chat_template_path and populate new metadata.
  • Added new unit tests for metadata population and registry immutability.
• tokenizer/src/tiktoken.rs
  • Implemented the backend() trait method for TiktokenTokenizer.
• tokenizer/src/traits.rs
  • Defined the backend() method in the Tokenizer trait.
  • Introduced the TokenizerBackend enum with HuggingFace, Tiktoken, and Mock variants, along with its Display implementation.

Activity

• No human activity (comments, reviews, etc.) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enriches the tokenizer metadata by adding backend, vocab_size, and created_at fields to TokenizerEntry and exposing them through the API. While the implementation is generally well-done, critical security concerns regarding Path Traversal and Information Exposure have been identified. As per repository rules, these cross-cutting security issues require dedicated design decisions and should be addressed in a separate, security-focused pull request. Additionally, consider improving the maintainability of the TokenizerBackend enum's string representation by providing an explicit Display implementation.

[slin1237]

Thanks for the work here — the metadata enrichment is useful and the implementation is clean. But I have a fundamental concern with the API design.

load() shouldn't accept metadata the registry can derive itself

The registry's job is: take an ID, name, source, and a loader function → load the tokenizer → store it. After loading, it should derive all metadata from the tokenizer object itself.

Right now chat_template_path is threaded through load() purely to store it as metadata — it's not used during loading at all (the loader closure already handles the actual template). This is the wrong direction:

// Current: caller supplies metadata TO the registry
registry.load(&id, &name, &source, chat_template_path, || async { ... })

// What it should be: registry loads, derives everything from the tokenizer
registry.load(&id, &name, &source, || async { ... })

vocab_size, backend, created_at — you're already deriving these from the tokenizer in TokenizerEntry::new(). That's correct. But chat_template_path breaks the pattern by requiring the caller to pass it in.

The fix

If chat_template_path is useful metadata, it should be queryable from the tokenizer object itself — either:

1. Add it to the Tokenizer trait: fn chat_template_path(&self) -> Option<&str>
2. Or store it internally in each tokenizer impl and expose it

Then the registry derives it the same way it derives vocab_size and backend — from the loaded tokenizer, not from the caller.

A tokenizer is a tokenizer. The registry shouldn't need to know about chat templates, vocab sizes, or backends as load parameters. It loads a tokenizer, then introspects it for metadata.

Other notes

• TokenizerBackend::Display uses {:?} (PascalCase: "HuggingFace") — most APIs use snake_case for enum values
• TokenizerEntry::new() should be pub(crate) not pub
• Missing trailing newline in Cargo.toml
• Consider Serialize/Deserialize on TokenizerBackend directly instead of converting to String in handlers

[mergify]

Hi @ppraneth, this PR has merge conflicts that must be resolved before it can be merged. Please rebase your branch:

git fetch origin main
git rebase origin/main
# resolve any conflicts, then:
git push --force-with-lease

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@model_gateway/src/core/steps/tokenizer_registration.rs`:
- Line 108: Call-site is passing caller-supplied chat_template
(chat_template.as_deref()) into tokenizer_registry.load, which lets callers
override registry metadata; remove that argument and let load() derive
chat_template_path from the loaded tokenizer instance. Specifically, in
tokenizer_registration.rs replace the .load(&id, &name, &source,
chat_template.as_deref(), || { ... }) call with .load(&id, &name, &source, || {
... }) (i.e., drop the chat_template/chat_template_path argument) and ensure any
code expecting that parameter is adjusted so metadata is computed inside
tokenizer_registry.load or the provided factory closure using the tokenizer
trait methods instead of from the caller.

---

ℹ️ Review info

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1ca2286 and f98283f.

📒 Files selected for processing (1)

• model_gateway/src/core/steps/tokenizer_registration.rs

[coderabbitai]

🛠️ Refactor suggestion | 🟠 Major

Remove caller-supplied chat_template_path from registry metadata input.

At Line 108, chat_template.as_deref() is passed into tokenizer_registry.load(...). This makes registry metadata caller-driven and can drift from the actually loaded tokenizer state (notably on remote-fallback paths). Keep load() deriving metadata from the tokenizer instance/trait and avoid passing this field at the call site.

♻️ Suggested call-site direction

-            .load(&id, &name, &source, chat_template.as_deref(), || {
+            .load(&id, &name, &source, || {

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@model_gateway/src/core/steps/tokenizer_registration.rs` at line 108,
Call-site is passing caller-supplied chat_template (chat_template.as_deref())
into tokenizer_registry.load, which lets callers override registry metadata;
remove that argument and let load() derive chat_template_path from the loaded
tokenizer instance. Specifically, in tokenizer_registration.rs replace the
.load(&id, &name, &source, chat_template.as_deref(), || { ... }) call with
.load(&id, &name, &source, || { ... }) (i.e., drop the
chat_template/chat_template_path argument) and ensure any code expecting that
parameter is adjusted so metadata is computed inside tokenizer_registry.load or
the provided factory closure using the tokenizer trait methods instead of from
the caller.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f98283f321

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve chat template metadata only when it is actually applied

Passing chat_template.as_deref() into tokenizer_registry.load unconditionally records the requested template path even when local creation fails and the flow falls back to fetch_tokenizer_from_worker, which builds the tokenizer with None for chat templates. In that fallback scenario (common in IGW mode when the gateway cannot load locally), /v1/tokenizers and /v1/tokenizers/{id} will report a chat_template_path that was never used, so the new metadata is inaccurate.

Useful? React with 👍 / 👎.

[mergify]

Hi @ppraneth, this PR has merge conflicts that must be resolved before it can be merged. Please rebase your branch:

git fetch origin main
git rebase origin/main
# resolve any conflicts, then:
git push --force-with-lease

[mergify]

Hi @ppraneth, this PR has been inactive for 14 days. Please update it or close it if it's no longer needed.

Note: PRs inactive for more than 30 days will be automatically closed.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 14 days. It will be automatically closed if no further activity occurs within 16 days. Leave a comment if you feel this pull request should remain open. Thank you!

[github-actions]

This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it. Thank you!