model: add sarvam_moe architecture support

[sumitchatterjee13]

Add support for sarvam_moe architecture (sarvamai/sarvam-30b).

• SarvamMoEForCausalLM is a straightforward extension of BailingMoeForCausalLM (see vLLM PR #33942)
• 19 layers: 1 dense FFN + 18 MoE layers (128 routed experts, top-6, 1 shared expert)
• Sigmoid gating with zero-mean normalized expert bias, routed_scaling_factor=2.5
• Combined QKV projection with QK RMSNorm, GQA (64 heads, 4 KV heads)
• Tokenizer: SentencePiece BPE converted to GPT-2 byte-level encoding in the converter

Test plan

1. Conversion from HF safetensors to GGUF (bf16, 264 tensors)
2. Quantization: Q8_0, Q6_K, Q4_K_M
3. Inference tested on RTX 5090 with all quantizations
4. Model generates coherent output with correct tokenization

AI disclosure
AI tools were used in an assistive capacity for this contribution.

[CISC]

There seems to have been a bad merge, please clean up your PR.

[sumitchatterjee13]

There seems to have been a bad merge, please clean up your PR.

Cleaned up the PR, Thank you.

[Rudra-Khatri]

Really waiting for the official llama cpp support for Sarvam

[Ankk98]

I tried compiling this branch on my local (Strix Halo, linux, vulkan).

• Got Build failure: src/models/sarvam-moe.cpp “cannot convert … n_embd_head_v from type uint32_t (llama_hparams::)(uint32_t) const…”
• Cause: missing parentheses / treating hparams.n_embd_head_v and hparams.n_embd_head_k as members instead of functions.
• Fix:

hparams.n_embd_head_v -> hparams.n_embd_head_v()
hparams.n_embd_head_k -> hparams.n_embd_head_k()

• Verification: after applying that change, cmake --build build --config Release completed successfully.
• Ran at [ Prompt: 232.4 t/s | Generation: 76.0 t/s ]

Vulkan Instance Version: 1.4.341
Kernel: 6.19.7-200.fc43.x86_64


ggml_vulkan: 0 = Radeon 8050S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

build      : b8401-77cc1dce0
model      : sarvam-30B-Q4_K_M.gguf
modalities : text

[sumitchatterjee13]

Thanks for reporting this and providing the fix! You're right — n_embd_head_v and n_embd_head_k were changed from member variables to functions in upstream llama.cpp. I've pushed the fix to the PR.

Also nice to see it running on Strix Halo with Vulkan!

I tried compiling this branch on my local (Strix Halo, linux, vulkan).

• Got Build failure: src/models/sarvam-moe.cpp “cannot convert … n_embd_head_v from type uint32_t (llama_hparams::)(uint32_t) const…”
• Cause: missing parentheses / treating hparams.n_embd_head_v and hparams.n_embd_head_k as members instead of functions.
• Fix:

hparams.n_embd_head_v -> hparams.n_embd_head_v()
hparams.n_embd_head_k -> hparams.n_embd_head_k()

• Verification: after applying that change, cmake --build build --config Release completed successfully.
• Ran at [ Prompt: 232.4 t/s | Generation: 76.0 t/s ]

Vulkan Instance Version: 1.4.341
Kernel: 6.19.7-200.fc43.x86_64


ggml_vulkan: 0 = Radeon 8050S Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

build      : b8401-77cc1dce0
model      : sarvam-30B-Q4_K_M.gguf
modalities : text

Thanks for reporting this and providing the fix! You're right — n_embd_head_v and n_embd_head_k were changed from member variables to functions in upstream llama.cpp. I've pushed the fix to the PR.

Also nice to see it running on Strix Halo with Vulkan!

[Ankk98]

The updated code compiles and runs fine

[pkhara31]

@sumitchatterjee13 Does this PR support for quantization in Q2_K_S or Q2_K ?

[fahdad]

@ggerganov and @CISC what can we do to help this get merged in? are there other plans to support the Sarvam models?

[CISC]

@ggerganov and @CISC what can we do to help this get merged in? are there other plans to support the Sarvam models?

I will get to it eventually, you are number 321 in line. :)

In the mean time it's probably useful to rebase and look at some of the recent changes to master, I think some of them may have made some of the changes in this PR redundant.

[fahdad]

@sumitchatterjee13 how can i help? and just making sure that your implementation will also cover the official gguf that they have since released as well right? https://huggingface.co/sarvamai/sarvam-30b-gguf

[sumitchatterjee13]

@sumitchatterjee13 how can i help? and just making sure that your implementation will also cover the official gguf that they have since released as well right? https://huggingface.co/sarvamai/sarvam-30b-gguf

Hey @fahdad , thanks for offering to help!

To clarify: Sarvam's official GGUFs on sarvamai/sarvam-30b-gguf use the existing bailingmoe2 architecture (general.architecture = bailingmoe2 in the GGUF metadata), not a new arch. So those files will continue to load via the existing bailingmoe2 code path in llama.cpp — they don't need this PR.

This PR adds a dedicated sarvam_moe architecture that properly handles Sarvam-specific details:

Full RoPE (Sarvam uses full rotary; bailingmoe2 uses partial with partial_rotary_factor=0.5)
Zero-mean expert bias normalization (Sarvam-specific, done during conversion)
SentencePiece → GPT-2 byte-level tokenizer conversion for Sarvam's tokenizer
Explicit SIGMOID expert gating

[CISC]

Please rebase and adapt to refactorings in master.

[pkhara31]

Tokenizer parity is broken on Indic scripts (and silently wrong on English)

Built this PR (commit [git rev-parse --short HEAD on your llama.cpp tree]) on Windows MSVC, CPU-only.
Model: [file name of your sarvam_moe-arch GGUF, e.g. sarvam-30b-q2_k_s.gguf].
HF reference tokenizer: sarvamai/sarvam-30b.

The GGUF tokenizer disagrees with the HF tokenizer on every probe. Generation in any non-Latin script is byte-fallback garbage as a result.

English - leading-space convention disagrees with HF

text:  "Hello, how are you doing today?"
hf:    [9259, 236764,   1217,   659,   611,  3490,  3124]   (8 tokens; GPT-2 byte-level: leading space is part of next word)
gguf:  [9259, 236764, 236743,  7843, 236743,  733, 236743]  (13 tokens; whitespace emitted as standalone token id 236743)

236743 appears as a separate token between every word, and the "no-leading-space" variants of how/are/today (7843/733/3124) are used instead of the trained Ġhow/Ġare/Ġtoday (1217/659/3124).

This means the runtime pre-tokenizer is splitting on whitespace and emitting the space as its own token, instead of using the GPT-2 byte-level convention the converter advertises.

Indic - content tokens missing entirely

Hindi  (नमस्ते, आप कैसे हैं? एक छोटी कहानी सुनाओ।)
hf:    [226767, 236764,   1772,  11590,   1545,  ...]   (12 tokens)
gguf:  [236764, 236743, 236743, 236743, 236881,  ...)   (9 tokens)

Kannada (ನಮಸ್ಕಾರ! ಒಂದು ಚಿಕ್ಕ ಕಥೆ ಹೇಳಿ.)
hf:    [237148, 237357, 82535, 8130, 236888, ...]   (11 tokens)
gguf:  [236888, 236743, 236743, 236743, 236743, ...]  (6 tokens)

Same pattern for Tamil, Bengali, Marathi, Telugu, Gujarati, Punjabi, Malayalam, Odia. The Indic content tokens are gone; the GGUF tokenization collapses to [punct?, 236743, 236743, 236743, ...]. This corresponds 1:1 to the <0xC4><0x..> byte-fallback / repetitive-loop generation we see at inference for Indic prompts.

Verdict

• HF tokenizer round-trip: PASS for all 13 probes (including all 10 Indic).
• GGUF token-ID parity vs HF: FAIL for all 13 probes.

So the model was trained against tokenization X, and llama.cpp on this PR is feeding it tokenization Y.

Repro
Test script: [link or paste]. Run:

python test_sarvam_tokenizer.py --gguf <your.gguf> --llama-tokenize <build>/bin/llama-tokenize

With either of:

• a GGUF freshly produced by this PR's convert_hf_to_gguf.py for sarvam_moe
• the file in this report

Suspected location
The PR description states: "Tokenizer: SentencePiece BPE converted to GPT-2 byte-level encoding in the converter."
The disagreement between HF and GGUF is consistent with tokenizer.ggml.pre being set to a value the C++ side does not route to a GPT-2 byte-level pre-tokenizer (so whitespace is split off as its own token instead of being attached as a leading-space prefix to the next word). Worth double-checking:

• the value of tokenizer.ggml.pre written by the converter for sarvam_moe
• the routing in src/llama-vocab.* for that pre value
• whether bytes_to_unicode() (GPT-2 mapping) is being applied to merges before they're written

[sumitchatterjee13]

Tokenizer parity is broken on Indic scripts (and silently wrong on English)

Built this PR (commit [git rev-parse --short HEAD on your llama.cpp tree]) on Windows MSVC, CPU-only. Model: [file name of your sarvam_moe-arch GGUF, e.g. sarvam-30b-q2_k_s.gguf]. HF reference tokenizer: sarvamai/sarvam-30b.

The GGUF tokenizer disagrees with the HF tokenizer on every probe. Generation in any non-Latin script is byte-fallback garbage as a result.

English - leading-space convention disagrees with HF

text:  "Hello, how are you doing today?"
hf:    [9259, 236764,   1217,   659,   611,  3490,  3124]   (8 tokens; GPT-2 byte-level: leading space is part of next word)
gguf:  [9259, 236764, 236743,  7843, 236743,  733, 236743]  (13 tokens; whitespace emitted as standalone token id 236743)

236743 appears as a separate token between every word, and the "no-leading-space" variants of how/are/today (7843/733/3124) are used instead of the trained Ġhow/Ġare/Ġtoday (1217/659/3124).

This means the runtime pre-tokenizer is splitting on whitespace and emitting the space as its own token, instead of using the GPT-2 byte-level convention the converter advertises.

Indic - content tokens missing entirely

Hindi  (नमस्ते, आप कैसे हैं? एक छोटी कहानी सुनाओ।)
hf:    [226767, 236764,   1772,  11590,   1545,  ...]   (12 tokens)
gguf:  [236764, 236743, 236743, 236743, 236881,  ...)   (9 tokens)

Kannada (ನಮಸ್ಕಾರ! ಒಂದು ಚಿಕ್ಕ ಕಥೆ ಹೇಳಿ.)
hf:    [237148, 237357, 82535, 8130, 236888, ...]   (11 tokens)
gguf:  [236888, 236743, 236743, 236743, 236743, ...]  (6 tokens)

Same pattern for Tamil, Bengali, Marathi, Telugu, Gujarati, Punjabi, Malayalam, Odia. The Indic content tokens are gone; the GGUF tokenization collapses to [punct?, 236743, 236743, 236743, ...]. This corresponds 1:1 to the <0xC4><0x..> byte-fallback / repetitive-loop generation we see at inference for Indic prompts.

Verdict

• HF tokenizer round-trip: PASS for all 13 probes (including all 10 Indic).
• GGUF token-ID parity vs HF: FAIL for all 13 probes.

So the model was trained against tokenization X, and llama.cpp on this PR is feeding it tokenization Y.

Repro Test script: [link or paste]. Run:

python test_sarvam_tokenizer.py --gguf <your.gguf> --llama-tokenize <build>/bin/llama-tokenize

With either of:

• a GGUF freshly produced by this PR's convert_hf_to_gguf.py for sarvam_moe
• the file in this report

Suspected location The PR description states: "Tokenizer: SentencePiece BPE converted to GPT-2 byte-level encoding in the converter." The disagreement between HF and GGUF is consistent with tokenizer.ggml.pre being set to a value the C++ side does not route to a GPT-2 byte-level pre-tokenizer (so whitespace is split off as its own token instead of being attached as a leading-space prefix to the next word). Worth double-checking:

• the value of tokenizer.ggml.pre written by the converter for sarvam_moe
• the routing in src/llama-vocab.* for that pre value
• whether bytes_to_unicode() (GPT-2 mapping) is being applied to merges before they're written

Thanks @pkhara31, your report was excellent — pinpointed the exact issue and made the fix straightforward.

Root cause confirmed. Your "Suspected location" section was spot-on: tokenizer.ggml.pre = "sarvam-moe" was routing to a custom regex that didn't match any of the byte-level custom implementations in unicode.cpp, so it fell through to std::regex which doesn't handle Unicode word boundaries properly — and worse, my converter was force-converting SentencePiece tokens to GPT-2 byte-level encoding, which collided with byte-fallback tokens.

Fix (pushed in the latest commit):

Switched to SPM-style BPE handling, mirroring LLAMA_VOCAB_PRE_TYPE_GEMMA4:
escape_whitespaces = true — runtime replaces input spaces with ▁ (U+2581) before BPE
byte_encode = false — BPE works on raw UTF-8 (no GPT-2 byte mapping)
Pre-tokenizer regex: [^\\n]+|[\n]+ (split only on newlines)
Reverted the SP→GPT-2 conversion in convert_hf_to_gguf.py. The converter now uses _set_vocab_gpt2() and keeps tokens in their original SP form with ▁.
This matches what Sarvam's tokenizer actually does (normalizer: " " → ▁, then BPE on the whole string).

Verification. I ran a parity test across 22 official Indian languages + English + mixed-script + edge cases (50 probes total):

50/50 probes passed
All probes round-trip identically against the HF reference tokenizer — including the exact ones from your report (Hindi, Kannada, Tamil, Bengali, Marathi, Telugu, Gujarati, Punjabi, Malayalam, Odia, plus Assamese, Urdu, Sindhi, Nepali, Sanskrit, Maithili, Konkani, Manipuri, Bodo, Santali, Kashmiri, Dogri).

Updated GGUFs are on the HF repo (BF16 / Q8_0 / Q6_K / Q4_K_M). Inference now generates fluent Hindi instead of byte-fallback garbage.

Thanks again for the detailed bug report — really appreciated.

[sumitchatterjee13]

Please rebase and adapt to refactorings in master.

Done, Thank you

[CISC]

Remove no longer needed arch/model code.

[fahdad]

this calls for some clebration @sumitchatterjee13 ! kingfisher? or taj?

[sumitchatterjee13]

this calls for some clebration @sumitchatterjee13 ! kingfisher? or taj?

Thank you. @fahdad 😊

Eather is fine... 😁