Add TML Inkling architecture

[danielhanchen]

Adds support for the Inkling architecture, a Python safetensors-to-GGUF converter, the graph build, and the kernel changes needed for correct and deterministic inference.

1. Had to use int64_t on some ops since large MoEs would go out of index
2. Made a Flash Attention banded attention kernel
3. Checked logits, outputs and long context + needle in a haystack and others
4. And more - will enumerate later
5. Has audio + vision support
6. Testing: Runs GGUFs made at https://huggingface.co/unsloth/inkling-GGUF multimodal well

I tried to keep changes unbreaking - might need to redesign some parts later.

Used AI for kernels, but hand verified and checked everything carefully.

[ggml-gh-bot]

Hi @danielhanchen, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• Multiple backend changes in one PR: When adding support for a new model or feature, focus on CPU support only in the initial PR. Add support for other backends like CUDA in follow-up PRs. If you have a good reason to modify multiple backends in one PR, please explain it.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[jaholmesuk]

First: thank you for day-0 support, and it works. We ran inkling UD-Q4_K_XL (587GB) fully resident across two cloud VMs (4x A100 80GB each, 8 total) split over the RPC backend on ordinary Ethernet, and output is coherent and correct (it first-shot passed our hardest structured Terraform generation task). Text only, audio/vision untested here.

We also profiled it, and the data below may be useful for the "might need to redesign some parts later" list. Decode runs at roughly 25 percent of what the hardware gives comparable MoEs, and the evidence points at per-token graph rebuild and launch overhead, not compute.

Numbers (all this PR at ce16fff, same boxes, sustained single-stream, ctx 8192)

Config	Decode t/s	Note
Qwen3-235B UD-Q4, 4x A100 single box	53.2	matches master (53.7), so this PR does not regress existing archs
Qwen3-235B UD-Q4, 8x A100 over RPC	44.5	matches master (45.9), so RPC is healthy
GLM-5.2-744B UD-Q4, 8x A100 over RPC (master, same boxes, previous day)	24.0	the expected class for a ~40B-active MoE at Q4
inkling UD-Q4_K_XL, 8x A100 over RPC, fully resident	6.45	41B active, expected ~GLM-class
inkling UD-Q2_K_XL, 4x A100 single box, fully resident	20.65	isolates RPC from the equation

Decode is flat from empty context to 2.2k depth (6.2 to 6.6), so it is not KV/attention scaling. Single box, inkling runs at ~80 percent of the comparable resident MoE. Over RPC it drops to ~27 percent.

Where the time goes

Per token at 6.45 t/s = 155 ms. During decode:

• All 8 GPUs sit at 0 to 6 percent utilization.
• llama-server burns ~35 percent of one host core.
• The host pushes ~5.5 MB per token to the RPC worker (36 MB/s at 6.5 t/s). Qwen-235B on the same pair moves ~0.75 MB per token.
• The server log reports graphs reused = 0 on every request, on both topologies.
• sched_reserve reports graph nodes = 8985, graph splits = 23, plus a 35 MiB CPU compute buffer (a few ops fall to CPU).
• GGML_CUDA_DISABLE_GRAPHS=1 changes nothing (20.74 vs 20.65 single box), so CUDA graph capture is not engaging either way and every node pays a discrete launch.

perf record on llama-server during decode gives a maximally flat profile: 1,783 distinct symbols, top entry 3.5 percent. Named entries in the top 20: cudaStreamSynchronize / ggml_backend_cuda_synchronize, pthread_mutex_lock under cuLaunchKernel, ggml_backend_sched_split_graph, ggml_gallocr_alloc_graph, and ggml-rpc add_tensor + get_alloc_size (the whole graph re-serialized to the worker every token, which is the 5.5 MB).

So the shape looks like: a ~9k-node graph rebuilt, re-split, re-allocated, re-serialized and launched node by node every token while the GPUs idle. Our guess is the banded rel-bias / shortconv path defeats the llama-side graph reuse, and the node count does the rest. If graph reuse engages for this arch, the RPC number should improve several-fold on its own.

Happy to re-run any branch of this PR on the 2-node A100 cluster or the single box and report the same measurements. Full logs, perf.data and exact commands available on request.

[F3zz1k]

Tried this on Intel Arc (SYCL), 6x Arc Pro B70, IntelLLVM 2026.0.0, with unsloth/Inkling-Small-GGUF UD-Q4_K_S. It runs and output is correct, but SYCL has no kernel for GGML_OP_FLASH_ATTN_EXT_BANDED (CUDA and CPU only) so -fa is force-disabled and attention falls back to a full ubatch x ctx score matrix.

llama-bench, 6 cards: pp512 409 t/s | pp2048 395 t/s | tg64 16.8 t/s
ctx 262144 @ ubatch 1024: tries to allocate a 103 GiB compute buffer, fails
ctx 65536 @ ubatch 256: loads, /health ok, then returns empty responses
ctx 30976 @ ubatch 256: works (llama-fit-params picked it, -ts 9,7,7,7,7,6)
--cache-type-v q8_0: "quantized V cache requires flash_attn" - V must be f16

Throughput looks about right for this setup however the context is capped at ~31k instead of the 1M I'd expect.

[wowo101]

No prompt-cache prefix reuse on inkling — every request re-prefills the full prompt.

Ryzen AI Max+ 395 / gfx1151, ROCm 7.14, this PR at 3fd7901cb (b10301), unsloth/Inkling-Small-GGUF UD-IQ3_XXS, -fa on, f16 KV, -np 1 -c 65536 -ub 256.

Measured by prefill wall time (timings.prompt_ms), so it does not depend on how the server reports token counts:

request	prefill	prompt_n
raw /completion, A = 4000 tok (cold)	27.2 s	4000
raw /completion, B = A + 1500 tok, cache_prompt: true	39.8 s	5500
/v1/chat/completions turn 1	27.5 s	3924
turn 2, prior turn appended	27.7 s	3925
turn 2, with reasoning_content also re-sent	28.0 s	3952

A full re-prefill of B predicts ~37.5 s; prefix reuse predicts ~10.2 s. Observed 39.8 s. On the chat path turn 2 re-processes the entire prompt rather than the ~20 new tokens. Re-sending reasoning_content makes no difference, so it is not a template-rendering mismatch on the assistant turn.

Control — same binary, same flags, same -ub 256, Qwen3.5-4B-UD-Q4_K_XL instead:

request	prefill	prompt_n
raw B = A + 1500 tok	0.91 s	1504 (delta only)
chat turn 2	0.09 s	23

So prefix reuse works on this build and these flags; it does not happen for inkling.

Why it matters beyond a benchmark: in an agentic/tool-calling loop the context grows every turn and each turn re-prefills all of it. A 14-turn recall task here spent its entire 25-minute budget in prefill — per-turn cost climbing 55 s → 163 s — and never reached a final answer, while the same task on another model completes in ~14 minutes. Total thinking across those 14 turns was only ~2.3k characters, so this is prefill, not generation.

Speculation, offered as a starting point rather than a finding: Inkling declares sliding_window 512 with a per-layer pattern in which 35 of 42 layers are windowed. An interleaved-SWA cache makes prefix reuse considerably harder than the dense case, so that seems the likeliest place to look.

Happy to run further diagnostics on this hardware — I still have the build and weights in place.

[khimaros]

recommend banning @wowo101 -- they are posting a lot of clearly AI generated content to issues/PRs

No prompt-cache prefix reuse on inkling — every request re-prefills the full prompt.

[wowo101]

These are all outcomes of pretty lengthy evaluation sessions, reviewed and edited – just trying to be helpful by sharing data. Will stop posting them if not welcome/useful – just let me know, not need for disciplinary action.

[F3zz1k]

I recommend banning @khimaros for being toxic and unhelpful. Don't like AI, then maybe a project that literally lets you run AI locally just isn't for you. If you think someone posts too often, too large of replies or in unhelpful ways, a mature person would engage with them and encourage them to improve rather than chase away the very community that supports the project.

@wowo101 I've noticed allot of toxic people on here when it comes to that. Allot of people come wanting to get involved and end up leaving because of it. Don't take it personally, people like that have problems. In general though, I've noticed people tend to be more appreciative if you condense your results down as much as possible. Don't let bullies push you out of open source.

Cache is indeed broken and it's for a known problem involving recurrent state. I actually have a fix for the general case on my fork but I need to update it for inkling(Recurrent works, swa works, but I never fixed recurrent+swa). If you're interested I can let you know when I've got it working. I also include auto disk caching and prefix checkpoint features if that's of any use.

In short, cache is a known issue not specific to this PR. Lots of people have posted bug reports about it but none of the official project devs have been willing to tackle it and they aren't accepting PRs from new user or anyone who is using any amount of AI to generate code. Hence why I had to fork it.

[khimaros]

@F3zz1k i am a fan of AI in general, however, many open source projects are being overwhelmed by these automated bots. even the most recent response by @wowo101 has obvious GenAI tells. it was not written by a human. anyway, this is the wrong forum for debating, so this will be my last reply.

[GeoMaciolek]

many open source projects are being overwhelmed by these automated bots.

Very real, and it's a challenging situation.

the most recent response by @wowo101 has obvious GenAI tells. it was not written by a human.

I generally recommend against accusing people of this; when it's actually a human, it derails the conversation, and frankly it's stressful and upsetting to get accused of. Plus, it doesn't seem to make much of a difference on actual LLM posts.

I've gotten accused of being an LLM in a few places, because I bother to use formatting and full sentences. "You sound like an LLM." Nope - I'm 45; LLMs sound like me (and my cohort) because among other things, they were trained on tons of Reddit posts by people like me.

As for if @wowo101 used an LLM in that last response? If they did, it didn't do a very good job, because "just let me know, not need for disciplinary action" is a grammatical mess. (Not intended to be insulting, @wowo101, I'm just pointing out a human mistake. Maybe a usage quirk from a non-native English speaker, or an editing mistake, but there's no way an LLM would let something like that past. Again, no insult intended, the meaning was perfectly clear.) Sure, it's possible that other stuff was LLM-assisted, but this amount of debate and noise isn't helpful; I only point these things out in hopes of defusing this and helping curb some of the reactive finger-pointing. Most of us want the best for these projects; slop submissions are a real problem, but arguments and accusations in comment threads are a problem too.

[F3zz1k]

@wowo101 I got the vram/ram cache for inkling fixed on my fork (see F3zz1k@371c74a ). As I said earlier, I've also got a bunch of other caching patches including auto-caching to disk and multimodal caching on recurrent, which I just updated with latest upstream and latest version of this inkling patch if you want it all applied already (see https://github.com/F3zz1k/llama.cpp ).

I only tested the fix on my full fork but it should work off of main too.

[brywil]

I tested this with this and then again with the following fix for the caching bug: F3zz1k@371c74a And ran with 20 tok/s on mt GB10 using this quant: https://huggingface.co/unsloth/Inkling-Small-GGUF

The build worked fine for me and the cache fix also sped up my cached turns by about 5x. (works)

[MatioLK]

Performance data on gfx1151 (Strix Halo, ROCm) — decode degrades linearly with context depth.

Ran a depth sweep with llama-bench -d on Inkling-Small UD-IQ2_M (76.76 GiB), since CUDA-only testing may not surface this.

Setup: AMD Ryzen AI Max+ 395 (gfx1151), 128 GB LPDDR5-8000, Ubuntu 26.04 (in-tree amdgpu), ROCm 7.14.0 in container, built from 1e6f9e4 with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151.

test	t/s	ms/token
tg32	20.96	47.7
tg32 @ d4096	8.86	112.9
tg32 @ d16384	3.36	297.6
pp512	121.94	—
pp512 @ d4096	127.89	—
pp512 @ d16384	75.10	—

The decode slope is constant — roughly 15.5 us of added latency per token of context:

   0 -> 4096  :  +65.2 ms / 4096  = 15.9 us/token
4096 -> 16384 : +184.7 ms / 12288 = 15.0 us/token

So it's O(n), not O(n^2) — but the constant seems high for this architecture. With sliding_window=512 on most layers, I'd expect decode to stay closer to flat as depth grows, with only the global layers scaling. Rough estimate: even if all 42 layers read their full KV window, that's under 1 us/token at this memory bandwidth, so the cost looks compute-bound in the banded bias rather than bandwidth-bound.

Possibly GGML_OP_FLASH_ATTN_EXT_BANDED isn't exploiting rel_extent_swa to bound the work per layer? I haven't read the kernel closely enough to say.

Practical impact: a 19.8k-token prompt with ~2k output takes about 18.5 min end-to-end on this hardware.

Two other notes for anyone testing on Strix Halo:

• --load-mode none (or --no-mmap) is required — without it, loading stalls badly past ~64 GB (looks like Rocm slow model weight loading past ~64GB mark on Strix Halo (all fine with Vulkan) #15018).
• -ub 256 is needed to avoid the ???? output issue reported earlier in this thread; the sweep above uses it, which may account for the modest prefill numbers.

Happy to run additional configurations if useful.

[MatioLK]

Three findings from testing on gfx1151 (Strix Halo, ROCm) that may be useful — in particular, the ???? output issue does not appear to be a ubatch problem.

Setup: AMD Ryzen AI Max+ 395 (gfx1151), 128 GB LPDDR5-8000, Ubuntu 26.04 (in-tree amdgpu), ROCm 7.14.0 in a container, built from 1e6f9e4 with -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151. Model: Inkling-Small UD-IQ2_M.

---

1. ???? output is caused by n_slots > 1, not by ubatch size

The workaround discussed earlier in this thread (-ub 256) did not help in my case. What did:

-np 1 --no-kv-unified

Reproduction is trivial and does not need a long prompt — an 18-token request is enough:

llama-server -m Inkling-Small-UD-IQ2_M-00001-of-00003.gguf -ngl 99 -fa 1 -ub 256 --load-mode none -c 32768 --jinja

Default n_slots = 4, kv_unified = true:

{"role":"assistant","content":"????????????????????????"}

Same server, same build, same model, with -np 1 --no-kv-unified:

{"role":"assistant","content":"Hello! How can I help you today? 😊"}

Notes that may narrow it down:

• llama-cli was always clean, at any prompt length — it runs a single slot.
• Broken output occurs at 18 input tokens, so the ~260-token final chunk boundary reported earlier isn't the mechanism, at least not the only one.
• -fa 0 does not avoid it either; only slot count does.
• Generation speed is normal while producing ???? — the model is computing at full rate, just emitting garbage.

Since the architecture carries per-layer short convolution state (shortconv_kernel = 4), my guess is that this recurrent state isn't isolated per slot, so slots overwrite each other. I haven't read the implementation, so that's speculation — but the -np 1 fix is reproducible.

---

2. -fa 0 is ~3x faster than -fa 1 at depth on this backend

Depth sweep with llama-bench -d:

test	-fa 1	-fa 0
tg32	20.96	—
tg32 @ d4096	8.86	—
tg32 @ d16384	3.36	13.85
pp512 @ d16384	75.10	118.39

With FA on, the decode slope is roughly 15.5 us of added latency per token of context, and it is linear:

   0 -> 4096  :  +65.2 ms / 4096  = 15.9 us/token
4096 -> 16384 : +184.7 ms / 12288 = 15.0 us/token

So O(n), not O(n^2), but the constant looks high. With sliding_window = 512 on most layers I'd expect only the global layers to scale with depth. Rough estimate: even if all 42 layers read their full KV window each step, that's under 1 us/token at this memory bandwidth — so the cost looks compute-bound inside the banded bias rather than bandwidth-bound. Possibly GGML_OP_FLASH_ATTN_EXT_BANDED isn't using rel_extent_swa to bound per-layer work?

Real-world, 8421-token prompt via /v1/chat/completions:

-fa 1 : prompt 102.2 t/s, decode  5.36 t/s
-fa 0 : prompt 111.3 t/s, decode 16.04 t/s

---

3. But -fa 0 degrades output quality noticeably

This part is subjective, so treat it accordingly — but the difference was consistent and large enough to report.

Same 8421-token prompt, same seed-free sampler settings, same preset except flash-attn. Task: summarise a document into a structured factual account. With -fa 1 the output covered the central facts. With -fa 0 the same prompt produced output that:

• omitted five of the six key facts present in the input (including the one uncontested admission, the date of the event, and the named witness),
• was ~40% shorter in substance,
• in one run, invented a witness who does not appear in the input, along with a fabricated relationship and a paragraph of testimony,
• in the reasoning trace, hallucinated an input detail ("broken glass") that isn't in the prompt and then built the whole analysis around contradicting it.

I don't have a numerical benchmark for this — no perplexity runs — so it may be worth someone reproducing on CUDA before drawing conclusions. But if the non-FA path is numerically diverging rather than just being a different code path, that seems worth knowing, since -fa 0 is currently the only way to get usable decode speed at depth on this hardware.

---

Also, for anyone else testing on Strix Halo: --load-mode none (or --no-mmap) is required, otherwise loading stalls badly past ~64 GB — looks like #15018.

Happy to run additional configurations or provide full logs if useful.