Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lib/llm/src/tokenizers/tiktoken.rs
Original file line number Diff line number Diff line change
Expand Up @@ -150,10 +150,10 @@ fn detect_bpe_pattern(directory: &Path) -> Result<&'static str> {
})?;

match model_type.as_str() {
"kimi" | "kimi_k2" | "kimi_k25" => Ok(KIMI_PATTERN),
"kimi" | "kimi_k2" | "kimi_k25" | "deepseek_v3" => Ok(KIMI_PATTERN),
Comment thread
biswapanda marked this conversation as resolved.
Resolved
Comment thread
biswapanda marked this conversation as resolved.
Resolved
_ => Err(Error::msg(format!(
"Unsupported tiktoken model_type '{model_type}'. \
Currently supported: kimi, kimi_k2, kimi_k25. \
Currently supported: kimi, kimi_k2, kimi_k25, deepseek_v3. \
To add a new model type, extend detect_bpe_pattern() in tokenizers/tiktoken.rs \
with the appropriate BPE regex pattern. \
Alternatively, provide a tokenizer.json (HuggingFace format) instead."
Expand Down
Loading