feat(frontend): add basetenkenizer backend

[maheshrbapatu]

Overview:

Adds Baseten Tokenizer as a selectable Dynamo frontend tokenizer backend using BasetenTokenizer exposed by the current upstream dynamo-tokenizers 1.7.0. Operators can select it with --tokenizer basetenkenizer or DYN_TOKENIZER=basetenkenizer.

Details:

• Use BasetenTokenizer from the current upstream dynamo-tokenizers 1.7.0.
• Accept basetenkenizer through the Python CLI/environment configuration, JSON runtime configuration, and Rust string parsing.
• Construct the Baseten backend for supported tokenizer.json files while preserving HuggingFace fallback behavior.
• Preserve special tokens declared in sibling tokenizer_config.json files and wrap the selected backend with the existing L1 prefix cache.
• Add token-parity, special-token, cache, segmented-encoding, and fallback coverage.
• Document configuration, compatibility, caching, and fallback behavior.

Where should the reviewer start?

• lib/llm/src/model_card.rs for backend construction, caching, and fallback.
• lib/llm/src/local_model/runtime_config.rs for configuration parsing and precedence.
• lib/llm/tests/model_card.rs for parity, cache, special-token, segmented-encoding, and fallback coverage.

Related Issues

• Relates to [CONTRIBUTION REQUESTED] Add Basetenkenizer as a selectable frontend tokenizer backend #12325

Validation

• cargo check -p dynamo-llm --lib --locked
• cargo test -p dynamo-llm --lib tokenizer_backend --locked (4 passed)
• cargo test -p dynamo-llm --test model_card --locked (10 passed)
• cargo test -p dynamo-llm --test tokenizers --locked (22 passed)
• Ruff, Rust formatting, codespell, and git diff --check

End-to-end performance validation

AIPerf exercised all three Dynamo frontend tokenizer backends against the same live Qwen/Qwen3-8B vLLM worker.

Hardware and software

• NVIDIA GH200 480GB platform: one GH200 GPU (97,871 MiB HBM reported by nvidia-smi) and one 72-core NVIDIA Grace/Arm Neoverse-V2 CPU.
• aarch64 Linux 6.8.0-111-generic; NVIDIA driver 610.43.02.
• Performance data was collected from PR commit 337655f924 using dynamo-tokenizers 1.6.0. Current upstream provides 1.7.0; its BasetenTokenizer implementation is unchanged from 1.6.0, and the merged tree passed the correctness checks above.
• AIPerf 0.11.0; vLLM 0.22.1rc1.dev144+g25f921b00.

Workload and method

• Qwen/Qwen3-8B, BF16, tensor parallelism 1, max model length 131,072 with static YaRN, vLLM prefix caching enabled, and 8,192-token chunked prefill.
• Synthetic streaming chat requests; actual input sequence length 128,008-128,009 tokens after chat templating; output sequence length 1; concurrency 1.
• Dynamo's L1 tokenizer prefix cache disabled with DYN_TOKENIZER_CACHE=0, confirmed by zero tokenizer-cache hit and miss counters.
• Three balanced, interleaved rounds per backend and model KV-reuse condition, with four measured requests per round: 12 measurements per cell and 108/108 successful requests overall.
• Model KV-cache reuse conditions: 0%, approximately 90%, and approximately 99%.

Mean time to first token

Model KV reuse	HuggingFace/default	Fastokens	Basetenkenizer	Basetenkenizer vs HuggingFace
0%	10,246.2 ms	10,094.0 ms	10,132.8 ms	1.11% lower
~90%	1,945.8 ms	1,757.5 ms	1,768.8 ms	9.09% lower
~99%	488.2 ms	303.5 ms	310.8 ms	36.35% lower

Mean Dynamo frontend tokenization time

Measured from dynamo_frontend_tokenize_seconds:

Model KV reuse	HuggingFace/default	Fastokens	Basetenkenizer
0%	191.60 ms	33.59 ms	27.38 ms
~90%	199.06 ms	22.43 ms	21.97 ms
~99%	202.14 ms	27.24 ms	22.80 ms

In this setup, Basetenkenizer reduced measured frontend tokenization time by 85.7-89.0% relative to HuggingFace and 2.1-18.5% relative to Fastokens. End-to-end Basetenkenizer and Fastokens remained close: Fastokens mean TTFT was 0.4-2.4% lower in these runs. The 36.35% result is specifically Basetenkenizer versus HuggingFace at approximately 99% model KV-cache reuse; it is not a general across-workload speedup. With a cold model KV cache, GPU prefill dominated and the TTFT reduction versus HuggingFace was 1.11%.

---

Summary by CodeRabbit

• New Features

  • Added support for the basetenkenizer tokenizer backend.
  • Enabled selection through the --tokenizer option and DYN_TOKENIZER environment variable.
  • Added fallback to the default tokenizer when alternate backends cannot load a model.
  • Preserved special tokens and prefix-cache behavior with the new backend.

• Documentation

  • Updated tokenizer configuration, compatibility, troubleshooting, and backend reference documentation.

• Bug Fixes

  • Improved tokenizer loading resilience for unsupported or invalid model configurations.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019fb945-c004-722a-afc2-289b3ceb4f66.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 0 potential issues.

[coderabbitai]

Walkthrough

The change adds the basetenkenizer tokenizer backend across dependency configuration, frontend and runtime parsing, model-card loading with HuggingFace fallback, integration tests, and frontend documentation.

Changes

Basetenkenizer tokenizer support

Layer / File(s)	Summary
Backend configuration and CLI wiring Cargo.toml, components/src/dynamo/frontend/frontend_args.py, components/src/dynamo/frontend/tests/test_frontend_args.py, lib/llm/src/local_model/runtime_config.rs	Adds the backend enum and string parsing, accepts it through CLI and environment configuration, updates the tokenizer dependency, and tests valid and invalid values.
Tokenizer loading and fallback lib/llm/src/model_card.rs, lib/llm/tests/model_card.rs	Selects Basetenkenizer at runtime, preserves special-token metadata, falls back to HuggingFace when loading fails, and tests encoding, decoding, segmented encoding, caching, and fallback behavior.
Backend documentation docs/fern/components/frontend/Tokenizer.md, docs/fern/components/frontend/frontend-config-reference.mdx, docs/fern/features/tokenizer/README.md	Documents the new backend, accepted configuration values, compatibility behavior, fallback handling, and TikToken limitations.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: adding the basetenkenizer backend.
Description check	✅ Passed	The description covers the required sections and adds validation details, with only minor template formatting differences in Related Issues.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Fix failing CI checks

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

components/src/dynamo/frontend/frontend_args.py (1)

91-91: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use frozenset to satisfy Ruff RUF012 on the shared class attribute.

Static analysis flags _VALID_TOKENIZER_BACKENDS as a mutable class-level default (RUF012). Since it's only read via membership checks, switching to a frozenset fixes the lint with no import changes needed.

♻️ Proposed fix

-    _VALID_TOKENIZER_BACKENDS = {"default", "fastokens", "basetenkenizer"}
+    _VALID_TOKENIZER_BACKENDS = frozenset({"default", "fastokens", "basetenkenizer"})

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/frontend_args.py` at line 91, Update the
shared class attribute _VALID_TOKENIZER_BACKENDS to use frozenset instead of a
mutable set, preserving its existing tokenizer backend values and
membership-check behavior.

Source: Linters/SAST tools

lib/llm/src/model_card.rs (1)

1260-1307: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the duplicated alt-backend load/fallback logic.

The Fastokens and Basetenkenizer arms are structurally identical (UTF-8 path check → try load → log → fallback to wrap_hf(hf)), differing only in the concrete tokenizer type and log strings. Consider factoring this into a small helper closure/function parameterized by a loader FnOnce(&str) -> Result<Arc<dyn Tokenizer>, E> and a backend label, to avoid tripling this boilerplate if a third backend is added later.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/llm/src/model_card.rs` around lines 1260 - 1307, Extract the duplicated
alternate-backend loading flow from the Fastokens and Basetenkenizer arms into a
reusable local helper or function parameterized by the backend label and loader,
while preserving the UTF-8 path validation, success logging, failure logging,
and wrap_hf(hf) fallback behavior. Update both TokenizerBackend branches to use
the helper and retain their concrete loaders and labels.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/fern/components/frontend/Tokenizer.md`:
- Around line 24-28: Update the link in the “basetenkenizer Native Encoder and
Decoder” section to remove the invalid Baseten Tokenizer repository URL or
replace it with the correct public reference, while preserving the surrounding
backend description.

---

Nitpick comments:
In `@components/src/dynamo/frontend/frontend_args.py`:
- Line 91: Update the shared class attribute _VALID_TOKENIZER_BACKENDS to use
frozenset instead of a mutable set, preserving its existing tokenizer backend
values and membership-check behavior.

In `@lib/llm/src/model_card.rs`:
- Around line 1260-1307: Extract the duplicated alternate-backend loading flow
from the Fastokens and Basetenkenizer arms into a reusable local helper or
function parameterized by the backend label and loader, while preserving the
UTF-8 path validation, success logging, failure logging, and wrap_hf(hf)
fallback behavior. Update both TokenizerBackend branches to use the helper and
retain their concrete loaders and labels.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 691f014a-9e52-4080-9e37-cf5656ec3cfc

📥 Commits

Reviewing files that changed from the base of the PR and between ce54943 and 337655f.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (9)

• Cargo.toml
• components/src/dynamo/frontend/frontend_args.py
• components/src/dynamo/frontend/tests/test_frontend_args.py
• docs/fern/components/frontend/Tokenizer.md
• docs/fern/components/frontend/frontend-config-reference.mdx
• docs/fern/features/tokenizer/README.md
• lib/llm/src/local_model/runtime_config.rs
• lib/llm/src/model_card.rs
• lib/llm/tests/model_card.rs

[ishandhanani]

/ok-to-test e28d719