model: add Hy3 (hy_v3) support with MTP speculative decoding by satindergrewal · Pull Request #25395 · ggml-org/llama.cpp

satindergrewal · 2026-07-07T09:57:47Z

Overview

Adds support for Tencent's Hy3 (hy_v3 / HYV3ForCausalLM, 299B MoE, 80 layers + 1 MTP layer), including its multi-token-prediction head as a draft-mtp speculative target. Addresses #24702 and #22477.

LLM_ARCH_HY_V3: arch registration, hparams (incl. n_layer_nextn), tensor tables with the 6 NEXTN entries
src/models/hy-v3.cpp: base decoder graph (sigmoid router + expert bias MoE, ungated shared expert, q/k norms) plus the MTP/nextn graph wired into the existing step35-style KV filter split
Converter: conversion/hunyuan.py (HYV3ForCausalLM) with --mtp/--no-mtp, layer-80 nextn mapping, final_layernorm -> shared_head.norm
No new speculative driver code: the existing --spec-type draft-mtp machinery runs it unchanged

Validation (RTX 5090, temp 0, 300-token coding prompt, deterministic across reruns):

Config	tok/s	Draft acceptance
no spec	5.81-6.31	-
draft-mtp n_max=3, p_min=0.75	7.97 (+26-37%)	97.3% (178/183), mean len 3.31
draft-mtp n_max=3, p_min=0 (default)	4.84 (slower)	38.6%

Build clean (CUDA 12.8, SM120). Regression: a qwen3.5-MoE MTP model is unaffected. Community hy_v3 GGUFs already in the wild (block_count 81) load unmodified.

Additional information

Reviewer notes, honestly flagged:

--spec-draft-p-min 0.75 is effectively required for this model: its MTP head is trained single-depth (per-position acceptance 0.878/0.224/0.010), so the p_min=0 default makes speculation a net slowdown. Open to discussing whether a per-arch default belongs in code.
blk.N.exp_probs_b is stored suffix-less for compatibility with published GGUFs; flagging for naming review.
llm_type is left UNKNOWN for the 299B config.
Perplexity comparison against the HF/vLLM reference has not been run (weights are 598GB; contributor hardware is consumer-class). Coherence, load, and A/B gates are as above.
Chat template and tool-call format polish for this family is out of scope and can follow separately.

The hy_v3 base graph is ported from charlie12345's community fork (credited via Co-Authored-By on the base commit); MTP forward semantics were verified against vLLM's hy_v3_mtp.py.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES. Full transparency: this implementation was written predominantly by an AI system (Anthropic Claude, Fable 5) working under the direction of @satindergrewal, who commissioned, reviewed, tested the resulting builds on real hardware, and takes responsibility for the contribution. The port follows an existing human implementation (charlie12345's fork) and vLLM's reference semantics rather than inventing new design. We understand this project restricts predominantly AI-generated contributions and we defer to the maintainers' judgment: if this cannot be accepted under policy, the PR still serves as a documented, validated reference for whoever implements hy_v3 support by hand, and we are glad to assist them.

@satindergrewal

Adds Tencent Hunyuan 3 (HF architecture HYV3ForCausalLM, GGUF arch hy_v3): a MoE decoder stack with per-head Q/K RMSNorm, a sigmoid router with expert selection bias, an always-active ungated shared expert, and leading dense block(s) (first_k_dense_replace). The base implementation is ported from charlie12345's fork (https://github.com/charlie12345/ROCmFPX, src/models/hyv3.cpp), adapted to current mainline APIs (hparams.n_layer(), build_qkv, build_moe_ffn with fused gate_up + scale tensors, output_s). Note: blk.N.exp_probs_b is stored without a .bias suffix for compatibility with existing hy_v3 GGUFs produced by that fork. Implemented by Claude (Anthropic Fable 5), directed by Satinder Grewal (@satindergrewal). Co-Authored-By: charlie12345 <charlie12345@users.noreply.github.com>

@satindergrewal

Loads the appended NextN/MTP decoder block(s) (blk.<n_layer>..) and adds a LLM_GRAPH_TYPE_DECODER_MTP graph so hy_v3 GGUFs converted with nextn_predict_layers work with --spec-type draft-mtp. Semantics follow vLLM's hy_v3_mtp.py (HYV3MultiTokenPredictorLayer): enorm(embed) + hnorm(prev_hidden) -> concat -> eh_proj -> full hy_v3 decoder block -> final_layernorm (stored as nextn.shared_head_norm) -> shared LM head. The checkpoint carries no MTP embed_tokens or separate shared head; both fall back to the main model's tok_embd/lm_head, matching vLLM's shared-weight loading. Hidden-state chaining is post-final_layernorm, so t_h_nextn is exposed post-norm in both the main graph and the MTP graph (same convention as qwen35moe). The main KV cache excludes the MTP layers and the MTP draft context only contains them (same filter split as step35). Measured on Hy3 299B-A15B IQ1_M (MTP block at Q8_0), RTX 5090, -ngl 12, temp 0, 300 tokens: draft acceptance 85.8% (n_max=3, mean accepted length 2.11, per-position 0.878/0.224/0.010), decode 8.81 tok/s vs 7.89 tok/s baseline (+11.7%) CPU-bound; n_max=1 acceptance 93.3%. Implemented by Claude (Anthropic Fable 5), directed by Satinder Grewal (@satindergrewal).

@satindergrewal

Registers MODEL_ARCH.HY_V3 in gguf-py (arch name, tensor table incl the NEXTN_* draft-head tensors) and adds HYV3Model to conversion/hunyuan.py. The NextN/MTP block sits at model.layers.<num_hidden_layers>; block_count is extended to include it (Step35Model pattern, including the --mtp / --no-mtp toggles) and its trailing final_layernorm is renamed to shared_head.norm so the existing NEXTN_SHARED_HEAD_NORM mapping picks it up. Everything else (mlp.router.gate, mlp.expert_bias, mlp.shared_mlp.*) resolves through existing tensor_mapping entries. Vocab is GPT-2 BPE with the existing hunyuan-dense pre-tokenizer. Implemented by Claude (Anthropic Fable 5), directed by Satinder Grewal (@satindergrewal).

ggml-gh-bot · 2026-07-07T10:02:05Z

Hi @satindergrewal, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.
AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

pwilkin

Looks fine to me, waiting for @CISC to take a look.

pwilkin · 2026-07-07T11:28:08Z

I'll add parser support in a followup PR.

satindergrewal · 2026-07-07T12:13:00Z

@pwilkin Parser notes from running these quants in the wild, in case they save you time:

Reasoning is wrapped in <think:opensource> ... </think:opensource>, and the chat template prefills the opening tag in the generation prompt, so output starts mid-reasoning and only the closing tag ever appears in the stream (delimiter-style parsing needed, an all-or-nothing optional(open + reasoning + close) grammar will miss it).
Tool calls are tag-based with tagged key/value arguments:
<tool_calls:opensource> section, per call <tool_call:opensource>NAME<tool_sep:opensource> followed by <arg_key:opensource>K</arg_key:opensource> / <arg_value:opensource>V</arg_value:opensource> pairs, closed with </tool_call:opensource> and </tool_calls:opensource>.
The eos token <｜hy_eos:opensource｜> uses fullwidth bars, DeepSeek-style. Tokenizer metadata types it CONTROL correctly.
Reasoning effort is a template kwarg (reasoning_effort: no_think / low / high); the stock HF template uses .format() calls that strict Jinja engines reject, a cleaned template ships with the GGUFs.

Happy to test any branch against the published GGUFs (IQ2_M with the MTP head is the quickest to exercise).

pwilkin · 2026-07-07T12:21:55Z

@satindergrewal I've got my own branch with the parser support already done :)

pwilkin · 2026-07-07T12:22:26Z

Need to exclude the MTP arch from test-archs.

@satindergrewal

hy_v3 requires expert_feed_forward_length (the arch is MoE-only, same as step35), so the synthetic Dense variant failed to load: error loading model hyperparameters: key not found in model: hy_v3.expert_feed_forward_length Adding LLM_ARCH_HY_V3 to moe_mandatory() makes the test provide the expert KVs and skip the Dense variant, mirroring the existing STEP35 handling. Addresses the test-archs review note on the hy_v3 PR. The SWA-pattern branch STEP35 also sits in does not apply: hy_v3 never reads LLM_KV_ATTENTION_SLIDING_WINDOW_PATTERN. Implemented by Claude (Anthropic Fable 5), directed by Satinder Grewal (@satindergrewal).

satindergrewal · 2026-07-07T12:45:04Z

@pwilkin Fixed in 97181b7. The actual failure was the synthetic Dense variant: hy_v3's loader requires expert_feed_forward_length, so the arch belongs in moe_mandatory() alongside STEP35 (one line). Full test-archs suite now passes, 115 archs, no other arch's result changed. The line-188 sliding-window branch was considered and deliberately left alone since hy_v3 never reads that key (noted in the commit body).

satindergrewal · 2026-07-07T23:59:54Z

Just finished testing the IQ2_M quant on my MacBook Pro M3 Max 128GB, and the results seem worth noting here as an Apple-silicon datapoint.

Metal, IQ2_M quant of the 299B, with --spec-draft-p-min 0.75: draft acceptance 88.7% (94/106); 23.27 tok/s with MTP vs 23.21 without. Net neutral on Metal: the setup is memory-bandwidth-bound, so the q8 MTP head's reads offset the accepted drafts. The same file on CUDA shows +26 to 37%.

This supports making p_min 0.75 the recommended or default setting for this arch: without it MTP is a slowdown everywhere; with it, it is a win on CUDA and harmless on Metal.

pwilkin · 2026-07-08T06:45:17Z

Actually, there's no use delaying stuff, @satindergrewal , could you please cherry-pick the last two commits from https://github.com/pwilkin/llama.cpp/tree/hunyuan-v3 onto here? (the parser fixes)

Assisted-by: Claude Fable 5

…rser Also disambiguate per-call vs section end markers by comparing their occurrence counts in single vs parallel tool call renders. Assisted-by: Claude Fable 5

satindergrewal · 2026-07-08T06:48:34Z

@pwilkin Done. Both commits cherry-picked onto the branch with your authorship preserved (692d171 jinja str.format, 56142c5 autoparser separator detection + the in-tree Hy3 template). The str.format support is especially welcome: it retires the fixed-template workaround we were shipping alongside the GGUFs. Running our validation battery against the updated branch now (the same chat/tools/streaming probes on the published 299B quants, both CUDA and Metal); will report anything unexpected.

satindergrewal · 2026-07-08T06:59:40Z

Validation battery on the updated branch (56142c5), Hy3 299B IQ1_M, CPU, temp 0, run against BOTH the in-tree tencent-Hy3.jinja and the stock GGUF-embedded template with --jinja only:

Probe	in-tree template	stock GGUF template
reasoning modes (default / no_think / high)	PASS	PASS
eos / think-tag leakage	none, any field, any mode	none
non-parallel tools (HTTP + JSON args)	PASS, clean args	PASS
streaming tools (deltas + finish_reason)	PASS	PASS

The headline holds: with the str.format support, the stock template renders and parses unmodified (10,175 chars, .format() calls and all, zero template errors). We had previously carried a hand-fixed template alongside the GGUFs and a set of hardcoded parser workarounds on a fork; your generic approach covers every case those patched, including an eos-after-tool-section 500 we had fixed symptomatically. The cause-level fix is clearly the right one, and tool-call argument quality on a 1.7-bit quant is visibly better here than under our workarounds (grammar engagement seems stronger with the detected separator).

One residual observation, not hy_v3-affecting: in the json_native tool path, parallel_tool_calls=false does not gate one_or_more(single_tool_parser) the way the tag paths do; sloppy low-bit quants on json_native model families could emit a malformed second call that fails strict finalize. Happy to file separately if useful.

satindergrewal · 2026-07-08T08:24:34Z

Triaged the CI failures before touching any code; the evidence says both are environmental, not the diff. Could a maintainer re-run the failed jobs?

ubuntu x64 (5 tests SIGILL): master's own run 28918782275 (commit 4a7ee31, zero PR code) fails with the identical five-test ILLEGAL signature (test-llama-archs, test-thread-safety, test-opt, test-rope, test-col2im-1d; the last was added by master's HEAD). Only ggml-compute tests die, all parser tests pass: runner ISA/ccache mismatch pattern. Locally: all five pass, Release and ASAN+UBSAN.

windows x64-openblas (0xc0000409 in test-jinja): all five new str.format tests PASS in the failing job's own log; the fail-fast fires ~600 lines later inside the pre-existing hasher property test at frame exit. The other three Windows jobs were matrix-cancelled, not crashed, so this is a single non-reproduced observation. Local: line audit of the str.format scanner (all accesses bounds-checked), 50 consecutive ASAN+UBSAN runs of test-jinja clean, full sanitizer suite clean.

Local gate on the branch as pushed: ctest 52/52 PASSED (Release, x64). If Windows reproduces at the same spot on re-run, I will instrument and dig further.

YanissAmz · 2026-07-08T09:07:04Z

I generated GGUFs for Hy3 that were published before this PR and have been downloaded by community members — would an arch alias be feasible?

Context: the hy_v3 GGUFs already published in the wild were converted with the earlier community implementation, which registered the architecture string as hy-v3 (dash) instead of this PR's hy_v3 (underscore):

https://huggingface.co/YanissAmz/Hy3-295B-A21B-GGUF (imatrix quants + reusable imatrix)
vcruz305's Q3/Q4 uploads (same converter lineage)

Everything else is already compatible — I verified against this branch (56142c5):

check	result
tensor names / tables	identical (incl. the 6 `blk.80.nextn.*` entries)
metadata keys	identical modulo the arch prefix
load after metadata-only rename `hy-v3.` → `hy_v3.`	loads and runs unmodified
`--spec-type draft-mtp --spec-draft-p-min 0.75` on the renamed file (Strix Halo 128GB, Vulkan)	24.3 t/s vs 17.4 baseline (+40%), 91% draft acceptance at temp 0.9, with a q4_0 MTP head

So a one-line arch alias accepting hy-v3 alongside hy_v3 (or a note in the docs pointing to a metadata rename) would keep those existing downloads working on mainline. I've published a rename script in the repo above as a fallback either way.

I remain available to test whatever you need on my hardware.

pwilkin · 2026-07-08T10:38:25Z

Nah, we have this problem all the time - the early converters should just reconvert, keeping aliases is messy and breaks conventions.

However, the llama.cpp convention is hy-v3, not hy_v3, so you should probably change yours :)

satindergrewal · 2026-07-08T10:55:20Z

On the naming: worth noting the tree already has underscore precedents, including in this same family: hunyuan_vl and ernie4_5. hy_v3 also matches Tencent's own model_type string, and at this point every Hy3 GGUF published in the wild (three publishers, this PR's validation corpus included) carries hy_v3 in its metadata, so a rename orphans all of them for a cosmetic dash. My preference is to keep hy_v3. That said, if maintainers prefer the dash, I will comply and publish a small metadata-fix script so existing files can be patched rather than reconverted.

ngxson · 2026-07-13T11:33:32Z

so are we discussing about source file name or something else?

pwilkin · 2026-07-13T11:48:10Z

No, the architecture name - I was just unsure if we want to enforce one convention or we're fine with either.

ngxson · 2026-07-13T12:17:53Z

the architecture name

what about:

if other models use _, use _ here
if other models use -, use - here
if no existing patterns, we don't care

pwilkin · 2026-07-13T12:23:03Z

Aight, @satindergrewal you're good to go in that case :)

ngxson · 2026-07-13T12:24:54Z

#25395 (comment) is not yet resolved

Per review: the only special sequence is '{' directly followed by '}'. Anything else after '{' throws not_implemented_exception, which the test harness already skips in cpp mode. Deletes brace-escape handling, field extraction and kwarg filtering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

YanissAmz · 2026-07-13T14:02:36Z

Fwiw from the GGUF side: the quants already published (mine and the other repos) were converted with hy-v3 (dash), so landing on the dash convention also keeps the ~44k already-downloaded files loading once this is merged. If it goes _ instead no big deal, metadata-only rename is easy — but figured it's worth mentioning.

ngxson · 2026-07-13T19:35:15Z

I'll let you decide the dash/underscore naming before merging this

pwilkin · 2026-07-13T20:50:36Z

@satindergrewal your final call, then I'll merge either way.

satindergrewal · 2026-07-13T21:07:56Z

Final call: keeping hy_v3 as-is. It matches Tencent's own model_type (tencent/Hy3 config.json: "model_type": "hy_v3"), and the tree has no single convention here anyway (hunyuan-moe/hunyuan-dense use dashes, hunyuan_vl/ernie4_5 use underscores), so following the vendor string is the least surprising rule. For the dash-converted GGUFs already published, @YanissAmz's metadata rename script covers them without reconversion. Ready to merge from my side. Thanks everyone.

std::stoi with a base outside {0, 2..36} trips the invalid parameter handler on the MSVC CRT (0xc0000409 fail-fast, not a catchable exception). The builtin-function fuzzer can generate such calls, e.g. '"s".int([1, 2, 3], -366)'; adding format() to the string builtins reshuffled the fuzz pairings and exposed this on the windows CI job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

satindergrewal · 2026-07-13T22:16:04Z

The windows x64-openblas CI red is fixed in a00f69a. Root-caused it fully because it turned out to be a latent engine bug, not something in this PR's features:

test-jinja was dying with 0xc0000409 (MSVC fail-fast, not a catchable exception) only on x64-openblas, which is the only Windows job that actually runs ctest.
Root cause: the jinja int() builtin passes an unvalidated base to std::stoi. The MSVC CRT fail-fasts on a base outside {0, 2..36} via its invalid parameter handler; glibc/libc++ return EINVAL instead, which is why Linux and macOS never crash.
Why this PR triggered it: the "fuzz builtin functions" test reflects over the builtins map, so adding format() changed builtins.size() and remapped every choice_dist(rng) % builtins.size() draw. The MSVC STL's distribution stream then rolled {{ "test_string".int([1, 2, 3], -366) }}, i.e. stoi with base -366. Deterministic under the fixed seed. Any future PR touching the builtins map could have tripped the same mine. (Side note for maintainers: uniform_int_distribution output differs between STL implementations, so the fuzz inputs already differ per platform despite seed 42.)
Fix: validate the base and throw a regular raised_exception. Verified on a windows-2025 runner with the exact x64-openblas job config before pushing: test-jinja fully green, full suite exit 0.

The remaining ubuntu x64 SIGILL failures are the pre-existing environmental issue triaged earlier in this thread (master and other PRs fail identically).

pwilkin · 2026-07-13T22:31:32Z

HF decided to act up, but the previous version already passed most tests, so merging.

tarruda · 2026-07-13T23:52:33Z

Is it possible to remove the MTP when converting to GGUF? When I try --no-mtp is says the option is only supported by Step or Qwen models.

Update: I've made a small refactor + enabled for Hy 3 here: #25641

…, OpenCL Q6_K/Adreno, CORS, checkpoint min-step, prompt cache refactor, MoE expert API stays) Upstream highlights since 6be7459: - model: DFlash speculative with KV rotation (ggml-org#25823) - model: Hy3 (hy_v3) with MTP speculative decoding (ggml-org#25395) - model: DeepseekV4 with fused hyper-connection ops (ggml-org#25585) - ggml: 0.17.0, LIGHTNING_INDEXER, out_prod, f16 set_rows - vulkan: Q2_0 support, native e2m1/e4m3 conversions, transfer-queue race fix - CUDA: MMQ kernel config refactor (ggml-org#24127), tighter MMQ src1 buffer for fp4 (ggml-org#25613), CUDA graphs on Volta/Turing, MoE gate/up dedup, CUDA Virtual Devices - ROCm: hexagon L2 cache rework, native fp4, FP16/INT8 coopmat on AMD - SYCL: Battlemage flash attention via oneDNN XMX, XIELU op, fp16 conv2d_dw - OpenCL: Q6_K GEMM/GEMV fix, ragged-tile MoE prefill FP16, Adreno vectorized LD/ST, A7x optimizations, ABS op - kleidiai: SME2 f32 kernel, SME vs SME2 dispatch - server: refactor prompt cache state ownership (ggml-org#25649) - new server_prompt_cache_state separates prompt metadata from KV data - server: evict checkpoints within min-step (ggml-org#25472) - server: text-only slot save/restore with mtmd (ggml-org#25076) - server: --cors-* options (ggml-org#25655) - server: refactored server_stream (ggml-org#25541) - server: respect min-step when splitting prompt batches (ggml-org#25420) - server: move chat-template thinking probe inside init try/catch (ggml-org#24093) - common: auto-download dflash/eagle3 HF sidecars (ggml-org#25811), drop --stdin mutual-exclusion, align tokenize usage - conversion: BitNetForCausalLM, dflash tokenizer fix, split MTP export for HY V3 - llama-quant: exclude i32 ffn_gate_tid2eid routing table, allow manual tensor types with --pure - llama-batch: fix allowed decreasing pos in a seq (ggml-org#25449), n_keep_tail in split_equal for recurrent - llama: refactor fused ops (ggml-org#24646), TP fix for Phi3/Bert/Plamo2/3/ChatGLM - ui: agentic content UX, reasoning effort on mobile add sheet, MCP panel fixes, thinking menu fix - vendor: BoringSSL 0.20250713.0 - tests: actually exercise test-recurrent-state-rollback, ds_v4_hc sentinel init, export-graph-ops graceful exit CachyLLama preservation work (conflict resolution): 1. tools/server/server-task.h: Accept upstream's server_prompt refactor (no data member, clear() method). Move our t_last_used field from server_prompt to server_prompt_cache_state (where it now lives after the refactor). server_prompt_cache_state already has the size() method, so our old size() on server_prompt is no longer needed. 2. tools/server/server-context.cpp (create_checkpoint): Take upstream's min-step eviction pre-filter as the FIRST pass, then keep our existing highest-pos_min eviction as the capacity overflow fallback. These are complementary: min-step removes redundant checkpoints from the same task; highest-pos_min keeps the rec-window-friendly checkpoints when at cap. 3. tools/server/server-context.cpp (handle_completions_impl): Keep our std::vector<server_task> tasks batching for multi-prompt requests and per-user concurrency check, AND take upstream's res->set_req(&req) for spipe ownership transfer. 4. tools/server/server-task.cpp: Fix references to entry.tokens -> entry.prompt.tokens, entry.checkpoints -> entry.prompt.checkpoints, entry.n_tokens() -> entry.prompt.n_tokens(). Update find_eviction_candidate return type from list<server_prompt>::iterator to list<server_prompt_cache_state>::iterator. 5. ggml/src/ggml-cuda/mmq.cuh + new mmq-config-rdna3_5.cuh: Upstream's massive MMQ refactor moved per-architecture config into separate files but did NOT add RDNA3.5 (gfx1150/1/2/3, Strix Halo). Create mmq-config-rdna3_5.cuh (231 CASE entries) derived from rdna2 with nthreads=128 (4 warps) and I=48 (smaller X tile) matching our original Strix Halo tuning. Wire into both host and device dispatch paths before the RDNA4 / RDNA2 fallback. 6. README.md and AGENTS.md: Keep CachyLLama-specific links and project context where upstream added parallel content. Verified: - cmake --build builds clean (Release, CPU-only) - llama-server starts, --help shows all CachyLLama flags preserved: --cache-ssd-hot-ram, --cache-ssd-warm-ram, --cache-ssd-system-prompts, --cache-ssd-system-max-days, --cache-ssd-no-fsync, --cache-ssd-max-conversations, --max-concurrent-per-user - /expert-stats and /expert-tracking endpoints preserved - 55/58 tests pass; 3 failures unrelated to merge: - test-tokenizers-ggml-vocabs: missing model downloads - test-jinja-py: missing jinja2 Python module - test-quant-type-selection: snapshot mismatch on upstream's new MXFP4_MOE heuristic Custom CachyLLama files untouched (no upstream conflicts): - common/kv-ssd-cache.{cpp,h}, common/kv-ssd-posix.h, common/kv-ssd-system-cache.{cpp,h} - common/kv_page_manager.{cpp,h} - tools/server/server-context-page-manager.{cpp,h} - tools/server/server-context-ssd-cache.{cpp,h} - test_kv_page_manager.cpp, tests/test-ssd-cache-caps.cpp - STRIX_HALO_NOTES.md, docs/development/user-isolation-design.md - .github/workflows/build-cpu.yml, build-cuda-windows.yml, build-vulkan.yml

…g#25395) * model: add Hy3 (hy_v3) architecture support Adds Tencent Hunyuan 3 (HF architecture HYV3ForCausalLM, GGUF arch hy_v3): a MoE decoder stack with per-head Q/K RMSNorm, a sigmoid router with expert selection bias, an always-active ungated shared expert, and leading dense block(s) (first_k_dense_replace). The base implementation is ported from charlie12345's fork (https://github.com/charlie12345/ROCmFPX, src/models/hyv3.cpp), adapted to current mainline APIs (hparams.n_layer(), build_qkv, build_moe_ffn with fused gate_up + scale tensors, output_s). Note: blk.N.exp_probs_b is stored without a .bias suffix for compatibility with existing hy_v3 GGUFs produced by that fork. Co-Authored-By: charlie12345 <charlie12345@users.noreply.github.com> Co-authored-by: Piotr Wilkin <ilintar@gmail.com> Assisted-by: Claude Fable 5

claude and others added 3 commits July 7, 2026 20:37

pwilkin approved these changes Jul 7, 2026

View reviewed changes

github-actions Bot added model Model specific conversion labels Jul 7, 2026

github-actions Bot added the testing Everything test related label Jul 7, 2026

pwilkin marked this pull request as ready for review July 7, 2026 13:18

pwilkin requested review from CISC and JohannesGaessler as code owners July 7, 2026 13:18

pwilkin added 2 commits July 8, 2026 18:48

jinja : add str.format() support

692d171

Assisted-by: Claude Fable 5

chat : detect separator between function name and arguments in autopa…

56142c5

…rser Also disambiguate per-call vs section end markers by comparing their occurrence counts in single vs parallel tool call renders. Assisted-by: Claude Fable 5

satindergrewal requested review from a team and ggerganov as code owners July 8, 2026 06:48

github-actions Bot added the jinja parser Issues related to the jinja parser label Jul 8, 2026

pwilkin approved these changes Jul 13, 2026

View reviewed changes

ngxson approved these changes Jul 13, 2026

View reviewed changes

pwilkin approved these changes Jul 13, 2026

View reviewed changes

pwilkin merged commit 2969d6d into ggml-org:master Jul 13, 2026
7 of 27 checks passed

tarruda mentioned this pull request Jul 14, 2026

convert_hf_to_gguf.py: Refactor how --mtp / --no-mtp is enabled for a model #25641

Merged

jhsmith409 mentioned this pull request Jul 14, 2026

Eval bug: hy_v3 loads exp_probs_b with no .bias suffix — existing Hy3 GGUFs (AngelSlim, pre-merge quants) fail with "wrong number of tensors; expected 1298, got 1218" #25657

Closed

reeselevine mentioned this pull request Jul 20, 2026

ggml-webgpu: Fix WASM compilation with OpenMP #25943

Merged

satindergrewal mentioned this pull request Jul 22, 2026

model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2) #25980

Merged

vakst mentioned this pull request Jul 29, 2026

Eval bug: NextN/MTP tensors now load by default for existing GGUFs, no load-time opt-out (regression from #25980) #26290

Closed

Uh oh!

Conversation

satindergrewal commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented Jul 7, 2026

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

satindergrewal commented Jul 7, 2026

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

pwilkin commented Jul 7, 2026

Uh oh!

satindergrewal commented Jul 7, 2026

Uh oh!

satindergrewal commented Jul 7, 2026

Uh oh!

pwilkin commented Jul 8, 2026

Uh oh!

satindergrewal commented Jul 8, 2026

Uh oh!

satindergrewal commented Jul 8, 2026

Uh oh!

satindergrewal commented Jul 8, 2026

Uh oh!

YanissAmz commented Jul 8, 2026

Uh oh!

pwilkin commented Jul 8, 2026

Uh oh!

satindergrewal commented Jul 8, 2026

Uh oh!

ngxson commented Jul 13, 2026

Uh oh!

pwilkin commented Jul 13, 2026

Uh oh!

ngxson commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Jul 13, 2026

Uh oh!

ngxson commented Jul 13, 2026

Uh oh!

YanissAmz commented Jul 13, 2026

Uh oh!

ngxson commented Jul 13, 2026

Uh oh!

pwilkin commented Jul 13, 2026

Uh oh!

satindergrewal commented Jul 13, 2026

Uh oh!

satindergrewal commented Jul 13, 2026

Uh oh!

Uh oh!

pwilkin commented Jul 13, 2026

Uh oh!

tarruda commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

satindergrewal commented Jul 7, 2026 •

edited

Loading

ngxson commented Jul 13, 2026 •

edited

Loading

tarruda commented Jul 13, 2026 •

edited

Loading