feat: allow deepseek_v3 architecture to use Kimi's bpe pattern

[biswapanda]

Overview:

baseten-admin/Kimi-2.5-text-nvfp4-v3 model has model_type: "deepseek_v3" in its config.json because Kimi K2.5 is built on the DeepSeek V3 architecture. it still ships the Kimi tiktoken tokenizer file, so the KIMI_PATTERN BPE regex is the correct pattern to use.

No pure DeepSeek V3 model uses tiktoken.model files (they use tokenizer.json instead) so this match is safe.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes DIS-1517

Summary by CodeRabbit

• New Features
  • Added support for deepseek_v3 model type to the tokenization pipeline, enabling proper token encoding and decoding for this model variant.

[coderabbitai]

Walkthrough

Added support for the "deepseek_v3" model type in BPE pattern detection by expanding the model_type matching logic and configuring it to use the KIMI_PATTERN tokenizer pattern.

Changes

Cohort / File(s)	Summary
Model Type Support lib/llm/src/tokenizers/tiktoken.rs	Added "deepseek_v3" to supported model types in detect_bpe_pattern, mapped to return KIMI_PATTERN, and updated error messaging to reflect the new supported model.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐰✨ A rabbit hops with glee,
For deepseek_v3 joins the spree,
KIMI_PATTERN blooms so bright,
Tokenizing dreams through the night! 🌟

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title clearly and concisely describes the main change: allowing the deepseek_v3 architecture to use BPE tokenization, which matches the core modification in the changeset.
Description check	✅ Passed	The description covers the Overview and Related Issues sections from the template with sufficient detail, though the 'Details' and 'Where should the reviewer start?' sections are not clearly present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/llm/src/tokenizers/tiktoken.rs`:
- Line 153: Add a regression test verifying that the tokenizer mapping in
lib/llm/src/tokenizers/tiktoken.rs still returns KIMI_PATTERN for the new
"deepseek_v3" branch: create a unit test (e.g.,
test_deepseek_v3_maps_to_kimi_pattern) that exercises the match logic which
includes the arms "kimi" | "kimi_k2" | "kimi_k25" | "deepseek_v3" and asserts
the returned value equals KIMI_PATTERN (or that the tokenizer behaves
identically to the existing "kimi" cases); place the test alongside other
tiktoken tests so future edits will fail if "deepseek_v3" is removed or changed.

---

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 64574a3 and 734d407.

📒 Files selected for processing (1)

• lib/llm/src/tokenizers/tiktoken.rs