[GGUF FE] Native .gguf graph builder - #37421
Draft
mvafin wants to merge 5 commits into
Draft
Conversation
github-actions
Bot
added
category: Core
4 tasks
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
4 times, most recently
from
August 13, 2026 21:11
6b69da6 to
a2cc31b
Compare
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
from
August 13, 2026 22:31
a2cc31b to
c61e617
Compare
mvafin
changed the title
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
2 times, most recently
from
August 13, 2026 22:55
b2adc14 to
80e761d
Compare
…cture Groundwork on the existing frontend, all of it reachable through the GgmlOvDecoder path that master already ships. The native .gguf builder is a separate change and does not appear here. MakeStateful. The frontend always converts to a stateless graph -- every KV cache an explicit Parameter/Result pair, as optimum-intel exports -- and being stateful is the consumer's choice, registered as a DecoderTransformationExtension so it runs ahead of the built-in stateless lowering. The pass also takes over beam_idx: it is a beam-search index into an OpenVINO state with no ggml counterpart, so declaring it in a decoder would leave a consumer-less input on the stateless graph. Op translators. Keep the output port when handing a value between translators (taking .get_node_shared_ptr() silently resolved to output 0, which throws for multi-output ops such as TopK); keep the static head layout in permute op_case 4; drop the builder-only op_case numbering from RESHAPE and VIEW; give each attention Transpose its own order constant; make the graph valid under both the SDPA and PagedAttention layouts by deriving the leading dims rather than pinning them; share the TopK-indices construction between ARGSORT and TOP_K, and let TOP_K tolerate a dynamic k instead of throwing. Quantization. Support the Q2_0 (ternary) type used by the Bonsai family. Decoder interface. Drop get_model_weights, which nothing calls. Tests. Add an op-coverage gate so a newly registered op cannot ship without a conversion test, and check the activation translators against captured output from real ggml rather than a numpy reimplementation of the formula -- a numpy oracle can only confirm the formula the author already guessed, which is how the GELU_QUICK error survived. CI. Add a GGUF_FE component so frontend changes scope their own jobs. ov_gguf_frontend_tests: 137/137. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the second ingest path: instead of a live GgufDecoder handed over by
llama.cpp, read the .gguf container directly and build the transformer graph
per-architecture, emitting nodes in the GGML op vocabulary so both paths share
the same op translators. No llama.cpp dependency.
One generic dense-transformer builder covers the llama family, with the
per-architecture differences derived from the file itself (q/k norms, qkv
biases, rope frequency factors, scalar scales), so most new architectures of
that family are just a name in the supported list. MoE routing, muse-glimmer
and qwen35's hybrid Gated-DeltaNet stack need real code and have it.
Also here, because they only exist once a model can be read from a file:
* AdaptToGenAI, which rewrites the llama.cpp-style IO into the OpenVINO GenAI
LLMPipeline contract, and the tokenizer metadata the frontend attaches to
rt_info so GenAI can build a tokenizer without reopening the file. Neither
is reachable from the llama.cpp integration, which owns its own tokenizer
and applies its own stateful transformation.
* MakeStateful's recurrent-state rewrite, for the linear-attention states
qwen35 carries. Unlike a KV cache these have no token axis and no SetRows
write, so the pairing arrives from the decoder via rt_info.
* The Q4_K integer zero-point: the CPU compressed-FullyConnected fast path is
~2x slower with a fractional f16 zp, which dominates prefill.
* Per-architecture conversion tests over header-only fixtures generated in CI
from llama.cpp, and a model_hub suite that downloads real checkpoints.
The frontend stays out of model auto-detection, so core.read_model(".gguf")
does not resolve to it; consumers either link it directly or ask for it by name
via load_by_framework("gguf").
Validated on all 25 checkpoints in tests/model_hub_tests/gguf: 25/25 convert,
and generation through GenAI matches the per-architecture expectations recorded
in docs/supported_models.md. ov_gguf_frontend_tests: 138/138.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…acy debugging The frontend docs already cover architecture bring-up, accuracy debugging and the testing architecture, but nothing routes an agent to them, so that knowledge is re-derived from sources on every session. Add three thin skills that point at the existing documents: - ov-gguf-enable-op -> docs/how_to_add_op.md (new) - ov-gguf-add-architecture -> docs/adding_an_architecture.md, supported_models.md - ov-gguf-debug-accuracy -> docs/debugging_accuracy.md Op enablement was the one procedural gap with no document, so add docs/how_to_add_op.md for it. It covers only the procedure and defers the concepts to frontend_design.md: the file checklist including the test CMake source list that is not globbed, the NodeContext accessors, the op-coverage gate in test_op_coverage.cpp and the fact that a narrowing --gtest_filter silences it, the rule that non-trivial reference values come from a ggml oracle rather than hand-derived math, and the build flag the target needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… seam
The native .gguf builder was one 1822-line file holding a single anonymous-namespace
class with ~53 members, ~20 of them m_has_*/m_is_* feature flags, and a 560-line
build(). Every layer of the design was present but none was separable: the block
helpers read builder state implicitly (build_dense_ffn(p, ffn_norm, T) also read
m_n_embd, m_weights and four MoE flags), so nothing could be reused by a model that
does not have those hyperparameters.
Split it along the seams it already had:
graph_emitter add_op / add_input / add_weight + shape and type bookkeeping,
knowing nothing about transformers (ggml_context's role)
blocks/ common, ffn, attention, gated_delta_net, qkv_repack as free
functions over GraphEmitter with explicit parameters
decoder_config the detection constructor + per-layer accessors
arch/decoder_builder the topology, i.e. only the order a decoder is assembled in
arch_registry the accept list and the one property that cannot be derived
from the tensor table (RoPE mode)
Largest file is now 404 lines.
Deliberately NOT one file per architecture, unlike llama.cpp's src/models/*.cpp.
That layout exists there because each architecture enumerates its tensors by hand;
this builder derives them from the GGUF tensor table, so ~30 architectures cost zero
lines each. Porting it would mean writing 30 files to replace 30 entries in a set.
For non-decoder families, take instead the split llama.cpp makes between
llm_graph_context and clip_graph: add ModelBuilder as the polymorphic seam, one
subclass per FAMILY (a distinct graph shape), with architectures staying data inside
a family. detect_model_kind() classifies a file from clip.has_vision_encoder /
clip.has_audio_encoder before any hyperparameter is read, and config_from_meta
becomes decoder_config_from_meta, because every key it reads is prefixed with the LLM
architecture name and an mmproj file carries clip.* keys instead. Without that split
an mmproj file failed in the metadata reader with a missing-block_count error rather
than a diagnostic naming what it actually is. Adding mmproj is now additive: detect,
add a family config reader, subclass ModelBuilder, add one dispatch branch.
No functional change. Verified by converting every architecture fixture before and
after and comparing the full graph signature -- op count, op-type histogram,
parameters, results, sinks, and each node's friendly name, type, output element type
and partial shape -- all identical, and the reject diagnostics unchanged except for
the source file the message names.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Conversion silently applies two approximations that move the weights away from
what the .gguf actually stores, so a user comparing output against llama.cpp has
no way to know part of the difference is by construction:
- token_embd / output / Q6_K / Q5_K are requantized channel-wise to Q8_0_C
(one int8 scale per row, replacing the file's per-16/32 block scales);
- Q4_K carries an INTEGER u8 zero-point, which rounds each sub-block's min to
a multiple of its scale.
Both reproduce the llama.cpp ggml-openvino backend's weight pipeline, and the
second is a deliberate perf trade (it keeps the dequant foldable into an int8
MatMul, ~2x prefill). Neither is wrong; both are worth knowing about before
chasing a numerical difference.
Report each at most once per process. A model has thousands of affected weights
-- every Q4_K matmul -- so anything per-tensor would bury the log, and the
message describes a fixed property of the conversion strategy rather than of one
tensor. One flag per kind, so a file that hits both still reports both.
The notice goes to std::cerr rather than OPENVINO_WARN. OPENVINO_WARN expands to
a no-op unless the build defines ENABLE_OPENVINO_DEBUG, which cmake/features.cmake
defaults OFF, so in a shipped build it reaches nobody -- and reaching the user is
the entire point here.
Emitted where the approximation is performed, not where its parameters are
queried, which puts the zero-point notice in the parser (the native path rounds
at parse time) and both notices in the raw-bytes weight builder (the cgraph
path). Verified: fires exactly once on a Q4_K model, stays silent on Q4_0 and
Q8_0 models, and leaves every converted graph bit-identical.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
.ggufgraph builder: instead of a liveGgufDecoderhanded over by llama.cpp, read the container directly and build the transformer graph per architecture, emitting nodes in the GGML op vocabulary so both ingest paths share the same op translators. No llama.cpp dependency.muse-glimmerandqwen35's hybrid Gated-DeltaNet stack need real code and have it.AdaptToGenAIand the tokenizer metadata attached tort_info, so OpenVINO GenAI can drive the model and build a tokenizer without reopening the file. Neither is reachable from the llama.cpp integration, which owns its own tokenizer and applies its own stateful transformation.MakeStateful's recurrent-state rewrite, for the linear-attention statesqwen35carries. Unlike a KV cache these have no token axis and noSetRowswrite, so the pairing arrives from the decoder viart_info.FullyConnectedfast path is ~2x slower with a fractional f16 zp, which dominates prefill.model_hubsuite that downloads real checkpoints.core.read_model(".gguf")does not resolve to it; consumers either link it directly or ask for it by name viaload_by_framework("gguf").tests/model_hub_tests/gguf: 25/25 convert, and generation through GenAI matches the per-architecture expectations recorded indocs/supported_models.md.ov_gguf_frontend_tests: 138/138.Important
Stacked on #37435 (part 1). GitHub only allows a base branch that lives in this repo, so this PR still targets
masterand its diff therefore includes #37435's commit. Review only the second commit,[GGUF FE] Native .gguf graph builder. It will shrink to that automatically once #37435 merges.Note
The CPU-side
WidenGatherMatmulWeightstransformation that MoE expert weights need is intentionally not in this PR and will follow separately; MoE models therefore need that change to compile.Tickets:
AI Assistance: