Granite-Switch Architecture by barvhaim · Pull Request #25107 · ggml-org/llama.cpp

barvhaim · 2026-06-28T16:02:44Z

Overview

Adaptation of Granite-Switch model (https://huggingface.co/ibm-granite/granite-switch-4.1-3b-preview) for llama.cpp,
Today Granite-Switch supports HF and vLLM backends.
Granite-Switch OSS project: https://github.com/generative-computing/granite-switch
Converted model with this code: https://huggingface.co/barha/granite-switch-4.1-3b-preview-GGUF/blob/main/granite-switch-4.1-3b-preview-f16.gguf

The model is dense, not MoE. This value is only needed because ggml_mul_mat_id uses n_expert_used as the number of selected matrix IDs per token. For Granite Switch this is always one selected adapter slot per token.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, parity with existing backends + documentations, assisted by Claude Code harness

P.S, maintainers, I'm looking forward for your comments on how to progress with getting this supported arch, so pretty much opened that PR for getting feedbacks. thanks!

ggml-gh-bot · 2026-06-28T16:07:01Z

Hi @barvhaim, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

barvhaim · 2026-06-28T17:16:14Z

Hi @barvhaim, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

I already added before "AI usage disclosure"

gabe-l-hart

A bunch of thoughts that loosely fall into the following buckets:

Should we explore implementing the adapters themselves using the existing adapter mechanism so it's more extensible in the future?
Some restructuring in the conversion code to hopefully simplify a little and extend flexibility

@CISC I'm particularly curious your thoughts on reusing existing adapter mechanisms vs this implementation that reimplements adapters in just the right places for this model so that it doesn't need to change anything about the existing adapter code paths.

gabe-l-hart · 2026-06-29T15:44:05Z

@@ -123,6 +123,192 @@ def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iter
        yield from super().modify_tensors(data_torch, name, bid)


+@ModelBase.register("GraniteSwitchForCausalLM")
+class GraniteSwitchModel(GraniteMoeModel):
+    """Dense, all-attention Granite model with N embedded LoRA adapters selected


I know the current model is built on the Dense 4.1 bases. I believe the intent is to also release switch models for other architectures going forward (including MoE). I haven't made it through the review yet, but I think we'll want to try to make sure the switch functionality is loosely coupled to the core model so it can easily extend to other core model architectures.

with removing the hack, expert_count now follows num_local_experts from the config
(still need approval on shared code modification)

gabe-l-hart · 2026-06-29T15:53:53Z

+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+        # the model is dense; force the dense invariant in case the llama parent


The GraniteMoeModel inherits directly from GraniteModel and only overrides set_gguf_parameters to optionally add the sahred_intermedaite_size (GraniteMoeShared). I don't think these explicit overrides are needed and they make the architecture less flexible to future core model architectures. In fact, I think the best course of action may be to inherit from GraniteHybridModel which itself acts as a child of both GraniteModel and GraniteMoeModel depending on the presence of SSM layers and experts. Some of the other logic may get more complex, but it's worth trying.

Dropped the hardcoded add_expert_count(0), it's config-driven now. Kept one dense guard on expert_used_count: the model's config.json carries num_experts_per_tok: 2 because the upstream HF config (GraniteSwitchConfig extends transformers.GraniteMoeHybridConfig) sets it, so without the guard a dense model emits expert_used_count=2 with expert_count=0 and fails the loader assert.

On reparenting to GraniteHybridModel: I tried it, seems like it's the wrong base here. Its __init__ auto-downgrades the arch when there are no SSM layers, and since the config has num_experts_per_tok: 2 it rewrites graniteswitch → granitemoe. It'd also pull in Mamba2 machinery (hparam_prefixes, SSM params, mamba set_vocab) that a dense all-attention model doesn't use. GraniteMoeModel stays the cleaner parent, the flexibility comes from the config-driven expert count, not the base class.

gabe-l-hart · 2026-06-29T17:49:30Z

+    auto * inp_attn = build_attn_inp_kv();
+
+    // router attention: a single causal head whose K/V live in the KV cache at
+    // layer R, recovering the per-token adapter index in-graph. Only dim 0 carries


Ah, I see: you keep the KV cache for this layer at the end, not the beginning, so we don't have to shift everything else.

gabe-l-hart · 2026-06-29T17:55:41Z

+    ggml_tensor * Kcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd_kv, qkv->ne[1], qkv->nb[1], n_embd_q*ggml_element_size(qkv)));
+    ggml_tensor * Vcur = ggml_cont(ctx0, ggml_view_2d(ctx0, qkv, n_embd_kv, qkv->ne[1], qkv->nb[1], (n_embd_q + n_embd_kv)*ggml_element_size(qkv)));
+
+    Qcur = ggml_add(ctx0, Qcur, build_switched_lora_delta(sl.a_q, sl.b_q, cur, adapter_ids));


Ok, I think this is the core of the in-model adapter implementation point. If we explore the route of using the existing llama_lora_adapter infra, this would get deferred into the build_attn below automatically I think.

CISC · 2026-06-29T18:47:52Z

@CISC I'm particularly curious your thoughts on reusing existing adapter mechanisms vs this implementation that reimplements adapters in just the right places for this model so that it doesn't need to change anything about the existing adapter code paths.

We have avoided this path a couple of times in the past already, but I think it's long overdue to try and tackle this, I think it would be worth refactoring adapters to handle this, unfortunately the timing is bad for me as I am on forced vacation. :)

I'm sure @ngxson has some thoughts on this as well.

gabe-l-hart · 2026-06-29T19:55:45Z

unfortunately the timing is bad for me as I am on forced vacation. :)

Zero problem (I hope everything is ok!). @ngxson if you're eager, I'd love to hear your thoughts, but I'm also game to take a pass at this if you're swamped. I had started it already, but hadn't gotten much further since @barvhaim was able to get things working end-to-end.

CISC · 2026-06-29T20:44:59Z

unfortunately the timing is bad for me as I am on forced vacation. :)

Zero problem (I hope everything is ok!).

Oh yes, given the circumstances fantastic in fact. :) Long story short I'm between jobs, all will be revealed by August...

ngxson

Given recent models like deepseekv4 / minimax also have complicated cgraph and kv logic, I think it could be fine to add this one. We can accept it as long as the impl is mostly confined inside granite-switch.cpp

Things that I'm less comfortable about the PR:

Excessive AI-generated comments. Most comments are just noise and doesn't add much to my understanding of the code
Still a lot of hacks outside of granite-switch.cpp

gabe-l-hart

A few more thoughts/comments, especially around simplifying the added tensor names with suffixes.

gabe-l-hart · 2026-06-30T17:19:16Z

+    def set_gguf_parameters(self):
+        super().set_gguf_parameters()
+
+        # dense model: override any expert counts the llama parent emitted


I'm still not sure we want to explicitly override like this. The upstream parents should handle this correctly for dense or MoE target models, so this just unnecessarily limits to dense-only.

barvhaim · 2026-07-02T12:09:32Z

Given recent models like deepseekv4 / minimax also have complicated cgraph and kv logic, I think it could be fine to add this one. We can accept it as long as the impl is mostly confined inside granite-switch.cpp

Things that I'm less comfortable about the PR:

Excessive AI-generated comments. Most comments are just noise and doesn't add much to my understanding of the code

Still a lot of hacks outside of granite-switch.cpp

+1
Removed the redundant AI generated comments

gabe-l-hart

One tiny NIT, but otherwise this looks like it's in good shape to me. @ngxson I think it's ready for CI and your next review when you have time.

gabe-l-hart · 2026-07-06T15:50:43Z

+// slots in dim 2 (slot 0 is zero-filled so base tokens add an exact-zero delta).
+// A: {n_embd_in, max_lora_rank, N}   B: {max_lora_rank, n_embd_out, N}
+// 7 injection sites (q, k, v, o, gate, up, down), each with an A and a B = 14 tensors.
+struct llama_layer_switch_lora {


Given @ngxson 's input, I think we're going to avoid the "overhaul how adapters work" piece for now.

ngxson

CI need to be fixed too

…g, TODO)

barvhaim · 2026-07-26T10:26:47Z

Quick note on this, running code-review reveals some correctness issues that must be address. You can get the skill here: #26042

+1

Update: All actionable code-review findings are now committed and pushed

ngxson · 2026-08-03T15:17:07Z

pinging @CISC again, appreciate if you can have a look on this PR this week

CISC · 2026-08-04T13:52:41Z

+    // extra single-head attention layer at the END (index n_real) holds the router
+    // K/V. reusing n_layer_nextn keeps n_layer() == n_real, so the regular layers
+    // keep their indices and the KV cache shift/defrag skips the router layer.
+    // n_layer_nextn is repurposed here (no MTP): it leaks as 1 into the
+    // llama_model_n_layer_nextn() getter and a re-saved nextn_predict_layers


Hmmm, this hack could have unintended sideeffects...

Is the worry about the shift/defrag logic?

It's just that n_layer_nextn is used a few places now, modifying it can mess things up unintentionally.

CISC · 2026-08-10T07:52:32Z

Sorry, forgot this one, let's merge.

gabe-l-hart · 2026-08-10T15:24:02Z

🎉 Thanks for all the help @ngxson @CISC !

28 upstream commits since 687e778. One touches ggml-opencl: 689e227 (ggml-org#26428, FA prefill K-tile transpose) -- our own PR, merged upstream this window, so its four files conflicted against the x2ue originals they were carved from. Resolved to ours in all four: x2ue is a strict superset (FA_Q_HALF / FA_O_HALF, the FA*_PROBE_NO_LDS diagnostics and FA_V_LDS_T all landed on top). Verified line-by-line that no ggml-org#26428 content was lost -- the seven upstream lines absent from the merged tree are the same code reshaped by our later commits (#elif rather than #if because our probe branch precedes, mad() rather than *, Q_PRIV_TO_ACC4() around q_priv), all still reading through FA_LK/FA_LK_PAIR, plus one reworded comment. Also verified the automerge kept upstream intact: of the 659 files upstream touched, the merged tree is byte-identical to upstream/master on all but the eight we also modified, and all 247 upstream-added lines in the four auto-merged files (clip.cpp, clip-model.h, llama-context.cpp, test-backend-ops.cpp -- Muse Glimmer ggml-org#26841, multi-output sampling ggml-org#25532, Granite-Switch ggml-org#25107) are present.

…, multi-output backend sampling, pocket-tts Merges 45 upstream commits (0865990..ebb546b, b10362-16-gebb546b7e) into synori/llama-update-mtp-fit. Zero conflicts; all vendored patches carried over untouched. Primary motivation — new Meta architecture: * 62bf73d model: Muse Glimmer Support (ggml-org#26841) LLM_ARCH_MUSE_GLIMMER + src/models/muse-glimmer.cpp + the mtmd vision tower in tools/mtmd/models/muse-glimmer.cpp and conversion/muse_glimmer.py. Other notable changes that touch our public API surface: * dd1ea52 llama : support multi-output backend sampling (ggml-org#25532) llama_context_params gains n_outputs_max_per_seq; llama_sampler_i.backend_init takes it as a third argument; new backend_reset / copy_state vtable slots and llama_sampler_copy(). * 153d324 llama : default load-mode auto, avoids mmap on iGPUs (ggml-org#26081) llama_load_mode gains LLAMA_LOAD_MODE_AUTO = -1 (enum is now signed). * 6e62ba5 mtmd: support pocket-tts (ggml-org#26871) mtmd_gen_inp/mtmd_gen_out gain seed/temp/feats/is_eos; new mtmd_gen_inp_default(); mtmd_helper_gen_audio_step_gen() takes out_stop. * 157b81f model : Granite-Switch Architecture (ggml-org#25107) * 7a20b41 model: MTP support for Nemotron (ggml-org#26725) and cc078b4 Dflash support for nemotron-3.5 (ggml-org#26905) * e23e944 vendor : cpp-httplib 0.53.0, 4c6766f vendor : subprocess.h sync Vendored patches preserved: * 919fde3 feat(rpc): thread-local last_error accessor — intact, upstream touched ggml-rpc.cpp by one unrelated line. * 3679b23 Fixes (RPC) — intact. * 2aa76c7 fix(metal): drop stray kernel_pad_f32 — still applies; upstream has since refactored pad into a templated kernel_pad_impl<T>, so the duplicate definition that referenced the nonexistent kargs_pad.s0..s3 is gone on both sides and nothing had to be re-applied.

* granite-switch: add llama.cpp backend (POC, CPU) New "granite-switch" architecture: a dense, all-attention Granite-4.1 model with N embedded LoRA adapters selected per-token by control tokens. - gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers - conversion/granite.py: GraniteSwitchModel converter (stacks N adapters + zero base slot into per-projection A/B tensors; emits switch metadata) - C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp}) - src/models/granite_switch.cpp: load + per-token switched-LoRA graph via ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token substitution in llm_graph_input_switch::set_input - llm_graph_input_switch in src/models/models.h Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13) and generate on both base and control-token paths. Sticky switch state is single-sequence (POC); full multi-sequence machinery is a follow-up. * granite-switch: add Mac (Metal) build + mid-sequence switch demo script Self-contained script to build llama.cpp on Apple Silicon (Metal), convert the composed 3b checkpoint, and run the crisp mid-sequence adapter-switch demos verified on Vela: - answerability: <|answerability|> mid-seq -> "unanswerable" - query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...} Each demo runs the same prompt twice, differing only by a control token placed before the assistant turn, so the per-token switch is visible. * granite-switch mac demo: add -no-cnv so each run is one-shot The composed model ships a chat template, so llama-completion auto-enables interactive conversation mode and halts at a `>` prompt after generating, stalling the script. -no-cnv disables conversation mode: generate once from the raw prompt and exit (also prints special tokens, making the switch visible). * granite-switch: replace global sticky index with in-graph router attention The POC computed the per-token adapter index on the CPU and carried it across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset only when a ubatch contained sequence position 0. That global had two problems: 1. Concurrency: with multiple sequences in a batch it was last-writer- wins — one sequence's adapter leaked into the others. 2. Multi-turn: an interactive `ollama run` chat continues one KV cache, so turn 2 never saw position 0 and the index never reset — the adapter stayed stuck on across turns. Port the vLLM/HF backend mechanism faithfully: a single-head causal "router" attention recovers the adapter index in-graph. Per token, only dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single visible control token recovers that adapter's slot (readback = clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is F16-safe (no F32 cache). The router's K/V live in the model KV cache at an extra layer R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1 so the cache allocator gives the router its own per-sequence slot, and set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and tensor loading are untouched and never reference layer R. The router K is exempted from the k-shift RoPE loop (its dim-0 value is a literal magnitude, not a rotation). Because the selection now lives in the per-sequence KV cache, CONCURRENT requests are isolated for free (problem 1 fixed; verified by scratch/concurrent_switch_test.cpp). set_input becomes stateless pure per-token maps; the global is gone. Single-switch contract / known limitation, identical to vLLM & HF: the gain is flat (no recency), so within one sequence there is no mechanism to revert to base mid-sequence — once an adapter fires it stays on until that sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF avoid it only because each served request is a fresh sequence). A client continuing one KV cache across turns must start a fresh sequence per turn, or opt into a recency-biased router (a deliberate divergence, not done here). Documented in granite_switch.cpp and asserted by scratch/multiturn_leak_test.cpp. Verified (CPU): both demos unchanged (answerability -> "unanswerable", query_rewrite -> rewritten query); concurrent two-sequence isolation passes; multi-turn carry-over matches the vLLM/HF contract. * granite-switch: drop scratch tests and mac demo for upstream PR Remove the local-only development artifacts that should not ship in the upstream PR: - granite-switch-mac-demo.sh (local Metal build + demo driver) - scratch/concurrent_switch_test.cpp - scratch/multiturn_leak_test.cpp Also drop the now-dangling reference to the scratch tests from the granite_switch.cpp header comment. Leaves only the core architecture support (conversion, gguf constants, llama-arch/model/kv-cache, and the granite_switch graph). * granite-switch: trim comments to match native llama.cpp style * granite-switch: trim conversion comments to match native style * granite-switch: drop unused adapter_ranks metadata * granite-switch: rename arch to graniteswitch and drop obid alias * granite-switch: fix non-ASCII comments and document router gain assumption * granite-switch: drop section comments from constants.py to match native style * granite-switch: add functional tensor block comments matching Granite4 Vision style * granite-switch: clarify n_expert_used comment State the actual constraint: mul_mat_id needs n_expert_used == 1, and since the GGUF carries expert_count = 0 the generic loader's n_expert == 0 => n_expert_used == 0 assertion has already passed by the time load_arch_hparams runs, so it is forced to 1 here. * granite-switch: note n_layer_nextn reuse has no MTP The router carving reuses n_layer_nextn, normally the MTP/next-token count. Clarify in the comment that it is borrowed here purely as the trailing-layers lever and that there is no MTP head, to spare readers the double-take. * granite-switch: rename source file and apply review nits * granite-switch: don't force LoRA tensors to F16, follow --outtype instead * granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly * granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0 * granite-switch: derive n_slots() * granite-switch: move llm_graph_input_switch into granite-switch.cpp * granite-switch: cut AI-style narration comments * granite-switch: collapse multi-line comments * granite-switch: rename control_token_* maps to adapter_token_* * granite-switch: cut noise comments * granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b * granite-switch: GGML_ASSERT token input to avoid UB on embeddings * granite-switch: TODO for raw embedding input support * granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix * granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe * granite-switch: stop forcing dense expert counts, read from config * granite-switch: renamed control_token_gain metadata key to router_gain * granite-switch: trim header comments to match native style * granite-switch: collapse LoRA tensors to base name + suffix * granite-switch: inline suffix checks in tensor op resolution * granite-switch: drop switch-lora struct comment * granite-switch: guard router layer index and inline n_slots * granite-switch: group adapter metadata under {arch}.adapters.* namespace * granite-switch: add hparams.has_rope(il) for KV-shift rope skipping * granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO) * granite-switch: Keys.Adapters namespace + simplify n_slots * granite-switch: validate substitute token ids against n_vocab * granite-switch: bound adapter count and lora rank from GGUF * granite-switch: reject MTP context type when router_layer is set * granite-switch: throw on bad adapter metadata instead of GGML_ASSERT * granite-switch: use ASCII +/- in router K signal comment * granite-switch: document n_layer_nextn repurpose and its leak points * granite-switch: gate lora_a/lora_b op mapping on router_layer * granite-switch: label all three preview model sizes

* feat(convert): Add conversion for GraniteSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(llama): Add granite_swa support Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add conversion infra for rope_pattern array NOTE: There is other work also targeting this, so this may be removed depending on merge order. Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(conversion): Fix SWA pattern logic and support for non-rope layers Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add support for GraniteMoeSWA Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add llama_hparams::has_rope and arch constants NOTE: This shadows the work done for Granite Speech #25107 Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add support for per-layer rope determination Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Fix failing flake8 for extra newlines Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver Branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Load MoE params as optional Branch: GraniteSWAForCausalLM AI-usage: draft (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Handle MoE params in conversion branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove unnecessary newline AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary tensor additions to GRANITE architecture Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Correctly handle naming for ffn gate inp Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Always default hparams.rope_pattern to 1s This isn't strictly necessary, but it will allow other models to rely on hparams.has_rope(il) without needting to prepopulate. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Move to has_rope for all granite model architectures Now that we have a proper hparam for this, it's better to use it and not require a hacky fallback in the hparam method itself. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: No hacky rope_finetuned fallback in has_rope Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fully remove rope hparam filling in granitemoe There are no granitemoe models that use NoPE (it's not actually used in the layer building below), so this was just dead code. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Save out rope_pattern in model-saver Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Set hparams.rope_finetuned for round trip Since the value is _read_ from rope_finetuned, we need to persist it when the model is saved with the saver. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Code review cleanup Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * refactor: Keep gate/up fused for MoE path Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Skip GRANITE_SWA in model saver #25505 (comment) Keeping is_swa_impl in the saver can break other models. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * add sliding window pattern for model in test * style: Fix indentation Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix \r\n Thanks Claude! Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Keep shared expert fused Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: More indentation fixes Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* feat(convert): Add conversion for GraniteSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(llama): Add granite_swa support Branch: GraniteSWAForCausalLM AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add conversion infra for rope_pattern array NOTE: There is other work also targeting this, so this may be removed depending on merge order. Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(conversion): Fix SWA pattern logic and support for non-rope layers Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(conversion): Add support for GraniteMoeSWA Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add llama_hparams::has_rope and arch constants NOTE: This shadows the work done for Granite Speech ggml-org/llama.cpp#25107 Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add support for per-layer rope determination Branch: GraniteSWAForCausalLM AI-usage: full (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Fix failing flake8 for extra newlines Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * test: Write out SLIDING_WINDOW_PATTERN in llama-model-saver Branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(convert): Fix missing registration for GraniteMoeSWAForCausalLM Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Load MoE params as optional Branch: GraniteSWAForCausalLM AI-usage: draft (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Handle MoE params in conversion branch: GraniteSWAForCausalLM AI-usage: full (OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove unnecessary newline AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary tensor additions to GRANITE architecture Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Correctly handle naming for ffn gate inp Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Always default hparams.rope_pattern to 1s This isn't strictly necessary, but it will allow other models to rely on hparams.has_rope(il) without needting to prepopulate. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Move to has_rope for all granite model architectures Now that we have a proper hparam for this, it's better to use it and not require a hacky fallback in the hparam method itself. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: No hacky rope_finetuned fallback in has_rope Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fully remove rope hparam filling in granitemoe There are no granitemoe models that use NoPE (it's not actually used in the layer building below), so this was just dead code. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Save out rope_pattern in model-saver Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Set hparams.rope_finetuned for round trip Since the value is _read_ from rope_finetuned, we need to persist it when the model is saved with the saver. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Code review cleanup Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * refactor: Keep gate/up fused for MoE path Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Skip GRANITE_SWA in model saver ggml-org/llama.cpp#25505 (comment) Keeping is_swa_impl in the saver can break other models. Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * add sliding window pattern for model in test * style: Fix indentation Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix \r\n Thanks Claude! Branch: GraniteSWAForCausalLM AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Keep shared expert fused Branch: GraniteSWAForCausalLM AI-usage: full (Claude + Sonnet 5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: More indentation fixes Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

barvhaim requested review from CISC and ggerganov as code owners June 28, 2026 16:02

github-actions Bot added model Model specific conversion labels Jun 28, 2026

barvhaim marked this pull request as draft June 28, 2026 18:04

pwilkin marked this pull request as ready for review June 28, 2026 18:14

pwilkin marked this pull request as draft June 28, 2026 18:14

barvhaim marked this pull request as ready for review June 28, 2026 18:42

gabe-l-hart self-requested a review June 29, 2026 15:40

gabe-l-hart reviewed Jun 29, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 111d333 to 9249531 Compare June 30, 2026 06:53

ngxson reviewed Jun 30, 2026

View reviewed changes

gabe-l-hart reviewed Jun 30, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 758fbbb to aa441eb Compare July 1, 2026 14:17

gabe-l-hart approved these changes Jul 6, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from bae32b6 to 0a7e411 Compare July 7, 2026 07:08

barvhaim requested a review from ngxson July 7, 2026 21:34

ngxson reviewed Jul 7, 2026

View reviewed changes

Comment thread src/llama-kv-cache.cpp Outdated

ngxson reviewed Jul 7, 2026

View reviewed changes

barvhaim force-pushed the feature/granite-switch branch from 290d8d4 to 97700cf Compare July 8, 2026 16:43

barvhaim requested a review from ngxson July 8, 2026 18:22

gabe-l-hart mentioned this pull request Jul 9, 2026

model: GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM #25505

Merged

ngxson reviewed Jul 12, 2026

View reviewed changes

Comment thread gguf-py/gguf/constants.py Outdated

Comment thread src/models/granite-switch.cpp Outdated

barvhaim requested a review from JohannesGaessler as a code owner July 12, 2026 12:23

barvhaim added 8 commits July 26, 2026 11:15

granite-switch: collapse LoRA tensors to base name + suffix

46d882e

granite-switch: inline suffix checks in tensor op resolution

d99296f

granite-switch: drop switch-lora struct comment

c57b00f

granite-switch: guard router layer index and inline n_slots

d2ca90b

granite-switch: group adapter metadata under {arch}.adapters.* namespace

11890d3

granite-switch: add hparams.has_rope(il) for KV-shift rope skipping

ae2ae69

granite-switch: skip arch in test-llama-archs (adapter fixture missin…

d886073

…g, TODO)

granite-switch: Keys.Adapters namespace + simplify n_slots

2204806

barvhaim force-pushed the feature/granite-switch branch from f57f905 to 2204806 Compare July 26, 2026 08:32

barvhaim added 8 commits July 26, 2026 11:55

granite-switch: validate substitute token ids against n_vocab

5669857

granite-switch: bound adapter count and lora rank from GGUF

a7271ce

granite-switch: reject MTP context type when router_layer is set

64278bb

granite-switch: throw on bad adapter metadata instead of GGML_ASSERT

34f5607

granite-switch: use ASCII +/- in router K signal comment

7cc8ba1

granite-switch: document n_layer_nextn repurpose and its leak points

118cf8a

granite-switch: gate lora_a/lora_b op mapping on router_layer

65cb201

granite-switch: label all three preview model sizes

15e473d

CISC approved these changes Aug 4, 2026

View reviewed changes

CISC merged commit 157b81f into ggml-org:master Aug 10, 2026
19 of 28 checks passed

Uh oh!

Conversation

barvhaim commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Requirements

Uh oh!

ggml-gh-bot Bot commented Jun 28, 2026

Uh oh!

barvhaim commented Jun 28, 2026

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CISC commented Jun 29, 2026

Uh oh!

gabe-l-hart commented Jun 29, 2026

Uh oh!

CISC commented Jun 29, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barvhaim commented Jul 2, 2026

Uh oh!

gabe-l-hart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barvhaim commented Jun 28, 2026 •

edited

Loading