model: Granite4 Vision

[gabe-l-hart]

Overview

This PR adds support for the Granite4VisionForConditionalGeneration mtmd architecture. It specifically targets the following models:

• ibm-granite/granite-vision-4.1-4b
• ibm-granite/granite-4.0-3b-vision

Additional information

The Granite4Vision models leverage several key architectural patterns that have not been previously supported:

1. Deepstack + Spatial projectors injected at non-contiguous points in the LLM layer stack
2. Llava-next encoder / assembler with learned newline token

Because of these two architectural patterns, this PR makes several key architectural shifts in the project:

Arch Changes in libllama

• llama_hparams.n_deepstack_layers -> llama_hparams.deepstack_layers_arr
  • This allows G4V to inject projector outputs at specific LLM layers
  • Backwards-compatibility is maintained for existing models using n_deepstack_layers, specifically the Qwen3VL family, by loading deepstack_layers_arr as either a single-valued number or a multi-valued array
  • Open Question: This type of try/catch backwards compatibility is not something I've seen elsewhere, so want to see whether this is a strong enough anti-pattern that I should instead just use a net-new hparam with overlapping meaning.

Arch Changes in mtmd

• Introduced a new class hierarchy for clip_assembler in clip-graph.h that parallels the clip_graph factory pattern
  • This class hierarchy will support models that have graph operations that need to happen after the individual image tiles have been encoded (eg llava-next style with learned newlines)
• Introduced clip_image_f32.append_token field that can be used by individual graphs to determine how to handle injecting learned newlines for each image tile.
• New public methods in clip.h to support the model-agnostic assembler logic
  • clip_image_assemble: This is the factory function for using the clip_assembler hierarchy to perform assembly
  • clip_n_assembled_output_tokens: This allows model-specific logic for counting output tokens based on how the assembly will work
• New hparam section for values that will be explicitly shared between an LLM and its MMPROJ
  • This is needed to bind the embedding_scale value to both the LLM and the MMPROJ so that the base stream can be pre-scaled to invert the embedding scaling that happens in the LLM
• ^ not needed anymore given skip logic for embedding scale w/ input embeddings
• clip_hparams.vision_feature_layer changed from an unordered_set to a vector to support a strict ordering and duplicate values. The ordering will map to the order of the projectors, multiple of which will pull from the same vision layer.
• Hoist QFormer tensors in clip_model into a qf_block struct and hold a vector of them in clip_model
  • This maintains backwards compatibility for Granite Speech which uses a vector of sized 1 while allowing G4V to support multiple blocks

Open Questions

Before merging, I want to address the following open questions:

• Maintainer alignment on introduction of clip_assembler paradigm Removed in favor of clip_image_f32.append_token
• Maintainer alignment on hparam try/catch single vs multi value parsing paradigm
• Mathematical alignment with alternate implementations (transformers and pure Claude implementation, see AI usage disclosure)
• Is there a cleaner way to skip the f_embedding_scale in llama-graph.cpp if (and only if) the input embeddings have valid image embeddings that doesn't require multimodal knowledge to leak into the core library?

Requirements

• I have read and agree with the contributing guidelines: YES
• AI usage disclosure: YES

AI Usage Disclosure

AI was used a lot in the creation of this PR! That said, the bulk of the work was actually meshing the AI's efforts into the existing architecture in a way that caused the least possible friction. I've annotated each commit with an AI-usage line (see stats below). There were two key ways that AI was used:

1. Granite Vision teammate @EliSchwartz built a working version of this branch entirely using Claude Code (here). This was heavily used as a reference implementation to check the implementation here that was more closely aligned with project patterns. Sections of this were referenced/copied verbatim (see commits with Co-authored-by).
2. Various agent/model combinations were used to assist in design/refactor throughout the branch

NOTE: I also failed with AI a bunch of times. Most agent/model combos couldn't handle the complexity of the architectural merger between G4V's architecture quirks and the various components of mtmd.

git-ai-stats

╔══════════════════════════════════════════════════════════╗
║ GIT AI USAGE ANALYSIS ║
╚══════════════════════════════════════════════════════════╝

📊 COMMITS BY AGENT

--- Aggregate ---
Commits | Count

none | 53
OpenCode + qwen3.5:122b | 5
Claude Code + Opus 4.7 | 4
IBM Bob | 1
Claude Code, IBM Bob | 1
OpenCode + Qwen 3.6-35B | 1
Claude Code | 1

TOTAL | 66

📊 COMMITS BY USAGE TYPE

--- Aggregate ---
Commits | Count

none | 53
draft | 6
full | 7

TOTAL | 66

📈 LINES OF CODE BY AGENT

--- Aggregate ---
Agent | Commits | Additions | Deletions

none | 53 | 1355 | 886
OpenCode + qwen3.5:122b | 5 | 50 | 3
Claude Code + Opus 4.7 | 4 | 463 | 296
IBM Bob | 1 | 13 | 0
Claude Code, IBM Bob | 1 | 600 | 7
OpenCode + Qwen 3.6-35B | 1 | 4 | 0
Claude Code | 1 | 30 | 0

TOTAL | 66 | 2515 | 1192

📈 LINES OF CODE BY USAGE TYPE

--- Aggregate ---
Usage Type | Commits | Additions | Deletions

none | 53 | 1355 | 886
draft | 6 | 802 | 27
full | 7 | 358 | 279

TOTAL | 66 | 2515 | 1192

[gabe-l-hart]

Ok, I think this is fully ready @ngxson. I found my two bugs that were causing the mathematical delta with Eli's version, so I've got exact matching output now!

[gabe-l-hart]

I think these test failures look unrelated since they're in test-backend-ops and this PR doesn't touch any kernels.

[gabe-l-hart]

@ngxson Gentle nudge. This PR should be ready for final review now.

[ngxson]

may worth checking upper bound for val too

[gabe-l-hart]

Actually, this is just counting the number of unique values, so I'm not sure this is the right place to guard against malicious values. That should probably be right above while loading (maybe just an assertion that the values are within the right range)

[gabe-l-hart]

Thanks for all the review help @ngxson !