model: add Mellum architecture

[Xarbirus]

Overview

This PR adds support for the new Mellum architecture (see hf).

Additional information

• It is important to note that the transformers version has been updated in this PR. This is because the converter does not work without the fix for one bug.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES for
  • analysis of current implementation
  • test execution and results analysis

[CISC]

Undo the formatting changes please. :)

Also fill in the AI disclosure in OP.

[CISC]

Sigh, old transformers limit huggingface_hub versions, try changing requirements/requirements-tool_bench.txt and tools/server/tests/requirements.txt lower requirement to >=0.34.0.

Edit: Scratch that, just remove huggingface_hub altogether, not sure why it's a dependency at all?

[Xarbirus]

Sigh, old transformers limit huggingface_hub versions, try changing requirements/requirements-tool_bench.txt and tools/server/tests/requirements.txt lower requirement to >=0.34.0.

Edit: Scratch that, just remove huggingface_hub altogether, not sure why it's a dependency at all?

Yep, I'll do that. I checked pyproject.toml, and everything seemed correct in there:( I'll remove the dependency from requirements.txt now.

[CISC]

Sigh, old transformers limit huggingface_hub versions, try changing requirements/requirements-tool_bench.txt and tools/server/tests/requirements.txt lower requirement to >=0.34.0.
Edit: Scratch that, just remove huggingface_hub altogether, not sure why it's a dependency at all?

Yep, I'll do that. I checked pyproject.toml, and everything seemed correct in there:( I'll remove the dependency from requirements.txt now.

Also in tools/server/tests/requirements.txt.

[g0t4]

383 tokens/sec generation on RTX 6000 Pro using Q8_0 from JetBrains/Mellum2-12B-A2.5B-Thinking

btw I replicated the conversion to GGUF f16 => Q8_0

nice work @Xarbirus

[CISC]

Failing CIs are not relevant, GTG.

[coder543]

Does this PR add support for Mellum2's MTP that is described in the Mellum2 paper?

[pilot7747]

@coder543 Not yet, will need to implement it in transformers first. But it's on the roadmap

[chris-hatton]

Can anyone report success using Tools with Mellum2 and llama.cpp?

Firstly; we need to apply the official Jinja template to get tools calling at all. But even then, 'Failed to parse' failures are frequent e.g:

"Failed to parse input at pos 239: <tool_call>\n{"name": "explore", "arguments": {"path": "/", "pattern": "App.js", "depth": 2}}\n</tool_call>"

The model is claimed as having been trained for Agentic operations; the high failure rate ~60% is therefore unexpected.

Update: Setting reasoning-format deepseek improves things further but still throwing Failed to parse input fairly often

My full llama-server command:

./llama.cpp/build/bin/llama-server \
  --model ./Models/Mellum2-12B-A2.5B-Thinking-Q6_K.gguf \
  --host 0.0.0.0 \
  -fa on \
  --ctx-size 100000 \
  --n-gpu-layers 999 \
  --ubatch-size 256 \
  --threads 8 \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --chat-template-file ./Templates/mellum2.jinja \
  --jinja \
  --reasoning-format deepseek

[Cyrille37]

Thanks @Xarbirus 💌

Works great with "Mellum2-12B-A2.5B-Thinking-Q6_K.gguf" on a RTX 3060 12Go with opencode.ai :

• prompt eval time = 9777.32 ms / 10596 tokens ( 0.92 ms per token, 1083.73 t/s)
• eval time = 26929.82 ms / 934 tokens ( 28.83 ms per token, 34.68 t/s)
• total time = 36707.14 ms / 11530 tokens

[chris-hatton]

Thanks @Xarbirus 💌

Works great with "Mellum2-12B-A2.5B-Thinking-Q6_K.gguf" on a RTX 3060 12Go with opencode.ai :

• prompt eval time = 9777.32 ms / 10596 tokens ( 0.92 ms per token, 1083.73 t/s)
• eval time = 26929.82 ms / 934 tokens ( 28.83 ms per token, 34.68 t/s)
• total time = 36707.14 ms / 11530 tokens

@Cyrille37 How is the tool calling for you? That's problematic for me, using the same named model file. Tools work but unusually high failure rate.

[Cyrille37]

How is the tool calling for you?

I only tried with opencode : tools like glob, grep, read, edit ... work fine.

[chris-hatton]

How is the tool calling for you?

I only tried with opencode : tools like glob, grep, read, edit ... work fine.

I'm also using OpenCode, are you willing to share your llama.cpp launch command for comparison? Mine's above.

[Cyrille37]

llama.cpp launch command

llama-server compiled with CUDA 12.9 capability 86

llama-server -m Mellum2-12B-A2.5B-Thinking-Q6_K.gguf \
    --host 0.0.0.0 --port 8012
    --verbosity 3 \
    --threads-http 2 \
    --no-mmap \
    --flash-attn on \
    -sm row \
    --cache-type-k q8_0 --cache-type-v q8_0 \
    --repeat-penalty 1.0 --presence-penalty 0.0 \
    --temp 0.2 \
    --jinja \
    --reasoning-format deepseek \
    -c 0

[Ar4l]

@chris-hatton we've uploaded a GGUF collection yesterday for which defaults should work for both tools and reasoning. But please let us know if the issue persists!

llama-server -hf JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q8_0 --tools all