Feature Request: Granite 4 Support

[gabe-l-hart]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

This issue is to track work to support IBM's Granite 4 model architecture (GraniteMoEHybrid in transformers). The model uses a number of components that are not yet supported in llama.cpp, but are being worked independently, so I'm raising this issue to triangulate the different work streams that will be needed to support the model.

Necessary Components

• Mamba2 layers
  • Ongoing work by @compilade: llama : initial Mamba-2 support #9126
• Refactored KV Cache to an abstract interface: kv-cache : separate recurrent vs non-recurrent impl #12799
• Support for hybrid attention / recurrent cache
  • Initial implementation for jamba by @compilade: llama : support Jamba hybrid Transformer-Mamba models #7531
  • Initial implementation for bamba: Bamba architecture #10810
  • Updated implementation for bamba that's also out-of-date: https://github.com/gabe-l-hart/llama.cpp/tree/BambaArchitectureRefactor
  • First cut implementation against current abstract interfaces: https://github.com/gabe-l-hart/llama.cpp/tree/HybridCache
• Support for GraniteMoEShared layers: Model: Granite MoE shared #13269
• Support for mamba2 in non-CPU backends
  • I'm not totally clear on the state here, so there may well be ongoing work
  • CUDA support for some of the necessary features was added in Faster ssm scan #10558
  • Some of the metal backend needs look like they're addressed already in llama : initial Mamba-2 support #9126, but for me that still doesn't work on my M3 (assertion error about non-contiguous data).
• Support for NoPE positional encoding instead of RoPE
  • I haven't fully investigated what is required for this, so it may already work as-is, but putting this here as a placeholder in case further work is needed
• End-to-end GraniteMoEHybrid support tying all of the other pieces together

Motivation

I lead IBM's efforts to ensure that Granite models work everywhere, and llama.cpp is a critical part of "everywhere!"

Possible Implementation

No response

[gabe-l-hart]

For reference, support PRs in other platforms:

• transformers: Add GraniteMoeHybrid support for 4.0 huggingface/transformers#37658
• vllm: [Model] Add GraniteMoeHybrid 4.0 model vllm-project/vllm#17497

[ngxson]

Support for NoPE positional encoding instead of RoPE

If this is the same idea with llama 4, then I think we already support this. In short, it's just an if condition:

llama.cpp/src/llama-model.cpp

Lines 4536 to 4547 in 3bf785f

	if (use_rope) {
	Qcur = ggml_rope_ext(
	ctx0, Qcur, inp_pos, rope_factors,
	n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
	ext_factor, attn_factor, beta_fast, beta_slow
	);
	Kcur = ggml_rope_ext(
	ctx0, Kcur, inp_pos, rope_factors,
	n_rot, rope_type, n_ctx_orig, freq_base, freq_scale,
	ext_factor, attn_factor, beta_fast, beta_slow
	);

[gabe-l-hart]

@ngxson That's great! Thanks for pointing that out

[gabe-l-hart]

Update: While not actually destined for merging, I decided the quickest path to get something running the model was to modify my previous branch for Bamba. I now have https://github.com/gabe-l-hart/llama.cpp/tree/GraniteFourDraft which is able to run a simple inference against the model and produce results. The code will need a pretty full rewrite once we get the overall hybrid architecture worked through, but this can be a starting point for anyone wanting to try the model and/or convert it to GGUF.

Full instructions for anyone that wants to try this out:

# Clone and build my branch
git clone git@github.com:gabe-l-hart/llama.cpp.git --branch GraniteFourDraft --depth 1
cd llama.cpp
mkdir build && cd build
cmake .. && make -j
cd -

# Download the model
huggingface-cli download ibm-granite/granite-4.0-tiny-preview --local-dir granite-4.0-tiny-preview

# Convert the model to GGUF
python convert_hf_to_gguf.py granite-4.0-tiny-preview

# (optional) Quantize
./build/bin/llama-quantize granite-4.0-tiny-preview/Granite-4.0-Tiny-Preview-62x915M-F16.gguf Q4_K_M

# Run the model
./build/bin/llama-cli -m granite-4.0-tiny-preview/ggml-model-Q4_K_M.gguf -ngl 0 -p "You are a helpful story teller. Tell me a story about a developer and their dog." -no-cnv

[gabe-l-hart]

For anyone following this issue, there's a mostly complete draft PR up: #13550

This is contingent on the following:

1. Hybrid Recurrent Cache: Hybrid recurrent cache #13979
2. mamba2: llama : initial Mamba-2 support #9126
   • Merge point of (1) and (2) being maintained at https://github.com/gabe-l-hart/llama.cpp/tree/mamba2-sync

[gabe-l-hart]

Support is officially merged! #13550