DiffusionGemma - #24423
Conversation
Some diffusion cli and visual updates
|
Hi @danielhanchen, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
Oof, that's a big one. There's a ton of debugging stuff left in there that needs throwing out, for one. I'm also not convinced about the idea to make a server just for one model - I think if we're intending to support diffusion models in a server mechanism, it should be a general diffusion-server (but that's just my opinion, probably have to wait for what @ggerganov thinks about this one). |
|
Haha sorry - this PR was more of a direct translation / proof of concept that it works! |
|
I'll edit the PR - sorry we're juggling multiple things haha |
|
Another PR for DiffusionGemma: #24427 |
|
You have some failing tests to fix. :) |
With a block diffusion model, couldn't the regular server just return each block when it is finished diffusing? It would be nice to just have one server, and the API could remain fully compatible so clients don't need to be aware that they're dealing with a diffusion model. (We don't show the distribution of sampled logits for AR models, and I don't see why people would need to see the intermediate diffusion steps either, since those won't be useful.) |
|
Doesn't build on Windows: |
|
Yep will fix haha - I also added a short GIF of it working edited in description! |
…s, drop debug hooks - guard sys/ioctl.h behind _WIN32 and add a GetConsoleScreenBufferInfo fallback for the visual viewport size, so diffusion-cli builds on Windows - skip diffusion-gemma in test-llama-archs like gemma4 (shared ISWA backbone, no synthetic fixture params yet) - remove the DG_DUMP_KV_LAYER / DG_NSWA debug scaffolding and its llama.h API - fix flake8 E306 in conversion/diffusion_gemma.py
|
I was able to compile it successfully on Linux for my 4090, but when running it, I get the following error after sending a user message: The command I'm running is: To compile it, I used: |
Builds and runs on Windows now. |
|
is it only cli at this point? no llama-server ? |
|
Thank you for putting this together! An Issue I found is that --fit doesn't work with this PR. |
|
Test run on my system with AMD hardware (7900 XTX) in it, Q4_K_M - time per step: 364.45ms Screencast_20260610_234032_c.webm |
|
@icedream What (equivalent) tokens per second are you getting? I also tried running it with an AMD GPU (R9700 w/ vulkan) and only got ~27t/s. |
|
@lucasbinder Not 100% sure if that's the right way to calculate it but based on two more runs with the same prompt, calculating with 256 tokens per full canvas diffused (I left out the last canvas as tail end of response), taking the start/end timings per canvas from the Run 1 2.327213 - 9.803998 = 7.476785 = 34.24 t/s Run 2 2.328457 - 9.777839 = 7.449382 = 34.37 t/s (Also I should clarify I used ROCm, not Vulkan in my case so that may be influencing the performance as well.) |
|
Unofficial prebuilt binaries for anyone who wants to test this PR without setting up a CUDA toolchain: https://github.com/gbuznote-beep/llama-diffusion-cli-prebuilt
Data points from testing (256 tokens, EB sampler): A5000 full-GPU 0.98 s/step; RTX 3070 Ti Laptop 8 GB via WSL2 ( |
|
Hey everybody, I missed the rule of no generated output with my previous comment. My bad, anyways I run a 9070xt. The gpu sampling switches back to the host when using Rocm cus line 665 only checks for cuda. If you add lookups for rocm and Musa you can get it to work and It gave me around 11% of a performance improvement. |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
|
looks like self promoting a paid service https://diffrun.dev/production-guide/ |
|
@rhjdvsgsgks thanks, hidden. |
Integriert DiffusionGemma (block text-diffusion MoE auf Gemma-4 Backbone) als monolithischen Port, da unser Fork die upstream llama_model_base- Hierarchie (Commit 994118a, 2026-05-04) noch nicht hat. Geaendert: - llama-arch.h/.cpp: LLM_ARCH_DIFFUSION_GEMMA, 5 neue Tensor-Typen, 7 Diffusion-KV-Keys (canvas_length, eb_*) - llama-model.h: Diffusion-spezifische Felder (canvas_length, sc_*, pkv_*) - llama-model.cpp: 4 Cases fuer load_hparams, load_tensors, build_graph, type_name Mapping - llama-model-loader.cpp: Template-Instanziierung get_key<int64_t> - include/llama.h: C-API Erweiterungen (diffusion_set_sc, set_phase) - models.h: Forward declaration llm_build_diffusion_gemma - gemma4-common.h: llm_graph_qkv Helper-Struct - diffusion-gemma.cpp: Monolithischer Graph-Builder (547 Zeilen) - tools/diffusion-cli: Minimales CLI-Tool fuer Tests - common/arg.cpp + common.h: CLI-Flags fuer Diffusion-Parameter - gguf-py: DiffusionGemma Arch/Tensor/KV-Mappings - tests/test-llama-archs.cpp: Arch in Skip-Listen eingetragen Neu (aus PR ggml-org#24423, unveraendert uebernommen): - conversion/diffusion_gemma.py: HF->GGUF Konvertierung - ggml-cuda/diffusion-sampling.cu: CUDA-Kernels fuer Diffusion-Sampling - pocs/: Dokumentation (Entscheidung, Masterplan, Status) Getestet: - Build erfolgreich (libllama.so, llama-server, llama-cli, llama-diffusion-cli) - Modell-Loading: Q4_K_M laeuft (25.25B Parameter) - Inferenz: Einzelner Forward-Pass mit greedy decode - GPU-Offloading: 8 Layers auf RTX 3070 Laptop (~2.5 t/s) Offen: - Echter Diffusion-Decoding-Loop (Prefill -> bidirektionaler Decode x N) - Entropy-bound Decoder - Visual Diffusion Mode - Upstream-Tools (diffusion-gemma-eval, -server) bauen nicht
Portiert aus PR ggml-org#24423 (Unsloth). Der Entropy-Bound Decoder ist der Kern-Algorithmus von DiffusionGemma: - Canvas mit zufaelligen Tokens initialisiert - Pro Step: Argmax, Entropie, Multinomial-Sample pro Position - Acceptance nach Entropie-Bound (niedrigste zuerst) - Renoising: nicht-akzeptierte Positionen -> frisches Rauschen - Output = Argmax-Canvas (nicht gesampelte Tokens) - Lineare Temperatur-Schedule (t_max -> t_min) - Konvergenz: Stabilitaet + Konfidenz Technisch funktionsfaehig. Output-Qualitaet verbesserbar durch: - Self-Conditioning (SC-Tensoren fehlen noch) - Mehr Steps (20 getestet, 48 empfohlen) - Chat-Template Prompt-Formatierung
|
It's so over for the GPU poor |
|
Looks like all checks passed, when is this getting merged? |
…vity, remove Spotify, fix cross-chat streaming bug, ship stats dashboard, land diffusion-gemma binary - Weather widget defaults to imperial (°F) instead of metric - Widget download button repositioned as a corner badge so it never overlaps card content (was covering the weather icon etc.) - Removed the Spotify embed widget entirely (tool, builder, dispatcher, UI) - Widget policy now tells the model to reach for a widget proactively whenever it's about to state a metric a widget covers, not just on request - Fixed app.streaming leaking across conversations: switching chats mid-reply no longer shows the other conversation's live thinking/response, and no longer corrupts the newly-viewed conversation's messages/title/context - New usage stats dashboard (StatsPanel.svelte): totals, per-model token bars, avg vs rolling-7d tok/s, full table — wired into Topbar/App - Compiled and deployed a diffusion-gemma-capable llama-diffusion-cli (unmerged ggml-org/llama.cpp#24423, Vulkan/RDNA4 build) and verified the full in-chat pipeline end-to-end; added extractFinalAnswer() to strip the CLI's <|channel>thought scratchpad and perf-stats trailer from saved replies
|
what are we waiting for ? any unresolved issues which are not discussed in this thread ? |
Refreshing the ggml-org#25731 pin was not enough on its own. It and ggml-org#24523 conflict with each other in common/chat.cpp and src/llama-arch.h: both append a new llm_arch value immediately before LLM_ARCH_UNKNOWN and both add a chat parser plus a detection block in the same regions. Whichever entry is applied second conflicts, and reordering does not help. #40 carries ggml-org#24523 with ggml-org#25731 merged into it and the conflict resolved, keeping every function and dispatch entry from both sides. It also adapts Inkling to the renamed thinking_end_tags API, so it builds against b10133 and newer. Replaying the resolver sequence on b10133 with these pins: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ a58a7fa6e OK unslothai #40 @ fee66f6a8 Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands.
ggml-org ggml-org#24523 now places LLM_ARCH_MINIMAX_M3 next to LLM_ARCH_MINIMAX_M2 instead of at the tail of the enum, matching how llama-arch.cpp already groups the name table. That removes the src/llama-arch.h half of the collision with ggml-org#25731, which appends LLM_ARCH_INKLING at the tail. ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags rename, so it now builds on its own against b10133 and newer. Both head SHAs moved, so the pins and the carrier branch are refreshed. unslothai #40 now only resolves what is left: both PRs still add a chat parser and a detection block in the same region of common/chat.cpp, immediately before "namespace workaround {". There is no principled alternative anchor there, since chat.cpp has no MiniMax-M2 parser to sit beside, so that half stays with the carrier. Replaying the resolver sequence on b10133: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ baee0f5 OK unslothai #40 @ 233cedb
#39 landed the first pin refresh only, so master still points at commits that have since moved on and that conflict with each other. ggml-org ggml-org#24523 was force-pushed: it now places LLM_ARCH_MINIMAX_M3 next to LLM_ARCH_MINIMAX_M2 rather than at the tail of the enum, matching how llama-arch.cpp already groups the name table. That removes the src/llama-arch.h half of its collision with ggml-org#25731, which appends LLM_ARCH_INKLING at the tail. New head baee0f5. ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags rename, so it now builds on its own against b10133 and newer. Its entry is replaced by unslothai #40, which carries what is still unresolvable between the two: both add a chat parser and a detection block in the same region of common/chat.cpp, immediately before "namespace workaround {". chat.cpp has no MiniMax-M2 parser to sit beside, so there is no principled alternative anchor for that half. Replaying the resolver sequence on b10133: OK ggml-org ggml-org#24423 @ c3fb972 OK ggml-org ggml-org#24523 @ baee0f5 OK unslothai #40 @ 233cedb Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands. Co-authored-by: Daniel Han <unslothai@gmail.com>
|
What's the status of this PR? All checks are passing, and I don't see any code issues to be addressed and acted upon? |
|
Thanks for still working on this even though it's been a while since weights dropped :D |
# Conflicts: # common/arg.cpp
Prebuilt: repin ggml-org#24423 and ggml-org#25731 onto carry branches
Worked on prelim Diffusion Gemma support!
llama-cliviallama-diffusion-cli -cnv -n 2048llama-diffusion-cli -cnv -n 2048 --diffusion-visualTo try this PR:
git clone https://github.com/ggml-org/llama.cpp cd llama.cpp gh pr checkout 24423 cmake -B build -DGGML_CUDA=ON cmake --build build -j --config Release --target llama-diffusion-clithen use a GGUF (any can work but for eg)
then use chat or visualization:
or
Example below (a bit blurry to limit to 10MB on Github :()

Disclaimer Heavy usage of AI, but verified logits matching with transformers, checked FP16 vs FP32 KV cache, long context checks and much more