Skip to content

DiffusionGemma - #24423

Draft
danielhanchen wants to merge 24 commits into
ggml-org:masterfrom
danielhanchen:diffusion-visual-updates
Draft

DiffusionGemma#24423
danielhanchen wants to merge 24 commits into
ggml-org:masterfrom
danielhanchen:diffusion-visual-updates

Conversation

@danielhanchen

@danielhanchen danielhanchen commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Worked on prelim Diffusion Gemma support!

  1. Has normal chat similar to llama-cli via llama-diffusion-cli -cnv -n 2048
  2. Has a visualization method to show diffusion live via llama-diffusion-cli -cnv -n 2048 --diffusion-visual

To try this PR:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
gh pr checkout 24423
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release --target llama-diffusion-cli

then use a GGUF (any can work but for eg)

pip install -U "huggingface_hub[cli]"
hf download unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --local-dir unsloth/diffusiongemma-26B-A4B-it-GGUF \
    --include "*Q8_0*" # Use "*Q4_K_M*" for a smaller 16 GB download

then use chat or visualization:

./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048

or

./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q8_0.gguf \
  -ngl 99 -cnv -n 2048 --diffusion-visual

Example below (a bit blurry to limit to 10MB on Github :()
diffusiongem-ezgif com-resize

Disclaimer Heavy usage of AI, but verified logits matching with transformers, checked FP16 vs FP32 KV cache, long context checks and much more

Some diffusion cli and visual updates
@danielhanchen
danielhanchen requested review from a team, CISC and am17an as code owners June 10, 2026 15:56
@github-actions github-actions Bot added model Model specific examples python python script changes labels Jun 10, 2026
@danielhanchen
danielhanchen marked this pull request as draft June 10, 2026 15:58
@ggml-gh-bot

ggml-gh-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

Hi @danielhanchen, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@danielhanchen danielhanchen changed the title diffusion-visual updates DiffusionGemma Jun 10, 2026
@pwilkin

pwilkin commented Jun 10, 2026

Copy link
Copy Markdown
Member

Oof, that's a big one.

There's a ton of debugging stuff left in there that needs throwing out, for one. I'm also not convinced about the idea to make a server just for one model - I think if we're intending to support diffusion models in a server mechanism, it should be a general diffusion-server (but that's just my opinion, probably have to wait for what @ggerganov thinks about this one).

@danielhanchen

Copy link
Copy Markdown
Contributor Author

Haha sorry - this PR was more of a direct translation / proof of concept that it works!

@danielhanchen

Copy link
Copy Markdown
Contributor Author

I'll edit the PR - sorry we're juggling multiple things haha

@gaugarg-nv

Copy link
Copy Markdown
Contributor

Another PR for DiffusionGemma: #24427

@CISC

CISC commented Jun 10, 2026

Copy link
Copy Markdown
Member

You have some failing tests to fix. :)

@coder543

Copy link
Copy Markdown
Contributor

if we're intending to support diffusion models in a server mechanism, it should be a general diffusion-server

With a block diffusion model, couldn't the regular server just return each block when it is finished diffusing? It would be nice to just have one server, and the API could remain fully compatible so clients don't need to be aware that they're dealing with a diffusion model. (We don't show the distribution of sampled logits for AR models, and I don't see why people would need to see the intermediate diffusion steps either, since those won't be useful.)

@quasar-of-mikus

Copy link
Copy Markdown

Doesn't build on Windows:

[421/433] Building CXX object examples\diffusion\CMakeFiles\llama-diffusion-cli.dir\diffusion-cli.cpp.obj
FAILED: examples/diffusion/CMakeFiles/llama-diffusion-cli.dir/diffusion-cli.cpp.obj
ccache C:\PROGRA~1\MICROS~3\2022\COMMUN~1\VC\Tools\Llvm\x64\bin\clang-cl.exe  /nologo -TP -DGGML_BACKEND_SHARED -DGGML_SHARED -DGGML_USE_CPU -DGGML_USE_CUDA -DLLAMA_SHARED -D_CRT_SECURE_NO_WARNINGS -IC:\Textgen\llama.cpp\src\..\include -IC:\Textgen\llama.cpp\ggml\src\..\include -IC:\Textgen\llama.cpp\common\. -IC:\Textgen\llama.cpp\common\..\vendor /DWIN32 /D_WINDOWS /EHsc /O2 /Ob2 /DNDEBUG -std:c++17 -MD /utf-8 /bigobj /showIncludes /Foexamples\diffusion\CMakeFiles\llama-diffusion-cli.dir\diffusion-cli.cpp.obj /Fdexamples\diffusion\CMakeFiles\llama-diffusion-cli.dir\ -c -- C:\Textgen\llama.cpp\examples\diffusion\diffusion-cli.cpp
C:\Textgen\llama.cpp\examples\diffusion\diffusion-cli.cpp(10,10): fatal error: 'sys/ioctl.h' file not found
   10 | #include <sys/ioctl.h>
      |          ^~~~~~~~~~~~~
1 error generated.

@danielhanchen

Copy link
Copy Markdown
Contributor Author

Yep will fix haha - I also added a short GIF of it working edited in description!

…s, drop debug hooks

- guard sys/ioctl.h behind _WIN32 and add a GetConsoleScreenBufferInfo fallback
  for the visual viewport size, so diffusion-cli builds on Windows
- skip diffusion-gemma in test-llama-archs like gemma4 (shared ISWA backbone,
  no synthetic fixture params yet)
- remove the DG_DUMP_KV_LAYER / DG_NSWA debug scaffolding and its llama.h API
- fix flake8 E306 in conversion/diffusion_gemma.py
@stepfunction83

stepfunction83 commented Jun 10, 2026

Copy link
Copy Markdown

I was able to compile it successfully on Linux for my 4090, but when running it, I get the following error after sending a user message:

0.13.483.785 E ggml_cuda_compute_forward: SOFT_MAX failed
0.13.483.796 E CUDA error: invalid argument
0.13.483.798 E   current device: 0, in function ggml_cuda_compute_forward at /home/LLM/DiffusionGemma/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:3163
0.13.483.798 /home/LLM/DiffusionGemma/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:103: CUDA error
E   err
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(+0x1c1ab)[0x7bb8567401ab]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7bb85674062c]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x15b)[0x7bb85674080b]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-cuda.so.0(_Z15ggml_cuda_errorPKcS0_S0_iS0_+0xb7)[0x7bb853260997]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-cuda.so.0(+0x27a810)[0x7bb85327a810]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x817)[0x7bb85675def7]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7bb855ee04c1]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114)[0x7bb855ee2c94]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6encodeERK11llama_batch+0x240)[0x7bb855ee6c80]
/home/LLM/DiffusionGemma/llama.cpp/build/bin/libllama.so.0(llama_decode+0xf)[0x7bb855eec1ef]
./build/bin/llama-diffusion-cli(+0x1941c)[0x5a66e697341c]
./build/bin/llama-diffusion-cli(+0x8143)[0x5a66e6962143]
./build/bin/llama-diffusion-cli(+0x60e4)[0x5a66e69600e4]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7bb85562a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7bb85562a28b]
./build/bin/llama-diffusion-cli(+0x6a45)[0x5a66e6960a45]
Aborted (core dumped)

The command I'm running is:

 CUDA_VISIBLE_DEVICES="" ./build/bin/llama-diffusion-cli \
  -m unsloth/diffusiongemma-26B-A4B-it-GGUF/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \
  -ngl 99 -cnv -n 2048 --system-prompt-file sysprompt.txt \
  --diffusion-eb auto \
  --diffusion-eb-max-steps 48 \
  --diffusion-eb-t-max 1.0 \
  --diffusion-eb-t-min 0.6 \
  --diffusion-visual

To compile it, I used:

rm -rf build
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_COMPILER=$(which nvcc)
cmake --build build -j --config Release --target llama-diffusion-cli

@quasar-of-mikus

Copy link
Copy Markdown

Yep will fix haha - I also added a short GIF of it working edited in description!

Builds and runs on Windows now.
1x 3090 Q4KM: time per step: 326.13ms
2x 3090 Q8_0: time per step: 878.83ms

@arkham000

Copy link
Copy Markdown

is it only cli at this point? no llama-server ?

@kroaton

kroaton commented Jun 10, 2026

Copy link
Copy Markdown

Thank you for putting this together! An Issue I found is that --fit doesn't work with this PR.

@icedream

icedream commented Jun 10, 2026

Copy link
Copy Markdown

Test run on my system with AMD hardware (7900 XTX) in it, Q4_K_M - time per step: 364.45ms

Screencast_20260610_234032_c.webm

@lucasbinder

Copy link
Copy Markdown

@icedream What (equivalent) tokens per second are you getting? I also tried running it with an AMD GPU (R9700 w/ vulkan) and only got ~27t/s.

@icedream

icedream commented Jun 10, 2026

Copy link
Copy Markdown

@lucasbinder Not 100% sure if that's the right way to calculate it but based on two more runs with the same prompt, calculating with 256 tokens per full canvas diffused (I left out the last canvas as tail end of response), taking the start/end timings per canvas from the --verbose output:

Run 1

2.327213 - 9.803998 = 7.476785 = 34.24 t/s
10.119182 - 17.995429 = 7.876247 = 32.50 t/s
18.412472 - 23.738067 = 5.325595 = 48.07 t/s
24.261660 - 32.754977 = 8.493317 = 30.14 t/s
33.345178 - 51.575057 = 18.229879 = 14.04 t/s

Run 2

2.328457 - 9.777839 = 7.449382 = 34.37 t/s
10.092776 - 17.935576 = 7.8428 = 32.64 t/s
18.349154 - 23.654470 = 5.305316 = 48.25 t/s
24.176041 - 32.655410 = 8.479369 = 30.19 t/s
33.245244 - 51.415716 = 18.170472 = 14.09 t/s

(Also I should clarify I used ROCm, not Vulkan in my case so that may be influencing the performance as well.)

@gbgh1

gbgh1 commented Jun 11, 2026

Copy link
Copy Markdown

Unofficial prebuilt binaries for anyone who wants to test this PR without setting up a CUDA toolchain:

https://github.com/gbuznote-beep/llama-diffusion-cli-prebuilt

  • Linux x86_64 / WSL2 — CUDA 12.8, sm_86 (RTX 30-series / A4000–A6000), glibc ≥ 2.39, self-contained (cudart/cublas/cublasLt/nccl bundled)
  • Windows x64 — CPU-only
  • Pinned to c84e85a; SHA256SUMS + reproducible build scripts included (other GPU archs rebuild in ~10 min)

Data points from testing (256 tokens, EB sampler): A5000 full-GPU 0.98 s/step; RTX 3070 Ti Laptop 8 GB via WSL2 (-ngl 99 --n-cpu-moe 22) 5.9 s/step; i7-12700H CPU-only 17.1 s/step. Output quality looks coherent (thinking-style drafts + self-critique). Thanks @danielhanchen for the implementation!

@CapCeph

CapCeph commented Jun 25, 2026

Copy link
Copy Markdown

Hey everybody, I missed the rule of no generated output with my previous comment.

My bad, anyways I run a 9070xt. The gpu sampling switches back to the host when using Rocm cus line 665 only checks for cuda.

If you add lookups for rocm and Musa you can get it to work and It gave me around 11% of a performance improvement.

@Jakeshadow

This comment was marked as spam.

@Jakeshadow

This comment was marked as spam.

@rhjdvsgsgks

Copy link
Copy Markdown
Contributor

https://diffrun.dev/status/

looks like self promoting a paid service https://diffrun.dev/production-guide/

@pwilkin

pwilkin commented Jul 3, 2026

Copy link
Copy Markdown
Member

@rhjdvsgsgks thanks, hidden.

fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
Integriert DiffusionGemma (block text-diffusion MoE auf Gemma-4 Backbone)
als monolithischen Port, da unser Fork die upstream llama_model_base-
Hierarchie (Commit 994118a, 2026-05-04) noch nicht hat.

Geaendert:
- llama-arch.h/.cpp: LLM_ARCH_DIFFUSION_GEMMA, 5 neue Tensor-Typen,
  7 Diffusion-KV-Keys (canvas_length, eb_*)
- llama-model.h: Diffusion-spezifische Felder (canvas_length, sc_*, pkv_*)
- llama-model.cpp: 4 Cases fuer load_hparams, load_tensors, build_graph,
  type_name Mapping
- llama-model-loader.cpp: Template-Instanziierung get_key<int64_t>
- include/llama.h: C-API Erweiterungen (diffusion_set_sc, set_phase)
- models.h: Forward declaration llm_build_diffusion_gemma
- gemma4-common.h: llm_graph_qkv Helper-Struct
- diffusion-gemma.cpp: Monolithischer Graph-Builder (547 Zeilen)
- tools/diffusion-cli: Minimales CLI-Tool fuer Tests
- common/arg.cpp + common.h: CLI-Flags fuer Diffusion-Parameter
- gguf-py: DiffusionGemma Arch/Tensor/KV-Mappings
- tests/test-llama-archs.cpp: Arch in Skip-Listen eingetragen

Neu (aus PR ggml-org#24423, unveraendert uebernommen):
- conversion/diffusion_gemma.py: HF->GGUF Konvertierung
- ggml-cuda/diffusion-sampling.cu: CUDA-Kernels fuer Diffusion-Sampling
- pocs/: Dokumentation (Entscheidung, Masterplan, Status)

Getestet:
- Build erfolgreich (libllama.so, llama-server, llama-cli,
  llama-diffusion-cli)
- Modell-Loading: Q4_K_M laeuft (25.25B Parameter)
- Inferenz: Einzelner Forward-Pass mit greedy decode
- GPU-Offloading: 8 Layers auf RTX 3070 Laptop (~2.5 t/s)

Offen:
- Echter Diffusion-Decoding-Loop (Prefill -> bidirektionaler Decode x N)
- Entropy-bound Decoder
- Visual Diffusion Mode
- Upstream-Tools (diffusion-gemma-eval, -server) bauen nicht
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
Portiert aus PR ggml-org#24423 (Unsloth). Der Entropy-Bound Decoder ist der
Kern-Algorithmus von DiffusionGemma:
- Canvas mit zufaelligen Tokens initialisiert
- Pro Step: Argmax, Entropie, Multinomial-Sample pro Position
- Acceptance nach Entropie-Bound (niedrigste zuerst)
- Renoising: nicht-akzeptierte Positionen -> frisches Rauschen
- Output = Argmax-Canvas (nicht gesampelte Tokens)
- Lineare Temperatur-Schedule (t_max -> t_min)
- Konvergenz: Stabilitaet + Konfidenz

Technisch funktionsfaehig. Output-Qualitaet verbesserbar durch:
- Self-Conditioning (SC-Tensoren fehlen noch)
- Mehr Steps (20 getestet, 48 empfohlen)
- Chat-Template Prompt-Formatierung
@Ninka-Rex

Copy link
Copy Markdown

It's so over for the GPU poor

@bhuv-z

bhuv-z commented Jul 11, 2026

Copy link
Copy Markdown

Looks like all checks passed, when is this getting merged?

CrankyVase added a commit to CrankyVase/duckpond that referenced this pull request Jul 13, 2026
…vity, remove Spotify, fix cross-chat streaming bug, ship stats dashboard, land diffusion-gemma binary

- Weather widget defaults to imperial (°F) instead of metric
- Widget download button repositioned as a corner badge so it never overlaps
  card content (was covering the weather icon etc.)
- Removed the Spotify embed widget entirely (tool, builder, dispatcher, UI)
- Widget policy now tells the model to reach for a widget proactively
  whenever it's about to state a metric a widget covers, not just on request
- Fixed app.streaming leaking across conversations: switching chats mid-reply
  no longer shows the other conversation's live thinking/response, and no
  longer corrupts the newly-viewed conversation's messages/title/context
- New usage stats dashboard (StatsPanel.svelte): totals, per-model token
  bars, avg vs rolling-7d tok/s, full table — wired into Topbar/App
- Compiled and deployed a diffusion-gemma-capable llama-diffusion-cli
  (unmerged ggml-org/llama.cpp#24423, Vulkan/RDNA4 build) and verified the
  full in-chat pipeline end-to-end; added extractFinalAnswer() to strip the
  CLI's <|channel>thought scratchpad and perf-stats trailer from saved replies
@corporatepiyush

Copy link
Copy Markdown

what are we waiting for ?

any unresolved issues which are not discussed in this thread ?

danielhanchen pushed a commit to unslothai/llama.cpp that referenced this pull request Jul 26, 2026
Refreshing the ggml-org#25731 pin was not enough on its own. It and ggml-org#24523
conflict with each other in common/chat.cpp and src/llama-arch.h: both
append a new llm_arch value immediately before LLM_ARCH_UNKNOWN and both
add a chat parser plus a detection block in the same regions. Whichever
entry is applied second conflicts, and reordering does not help.

#40 carries ggml-org#24523 with ggml-org#25731 merged into it and the
conflict resolved, keeping every function and dispatch entry from both
sides. It also adapts Inkling to the renamed thinking_end_tags API, so
it builds against b10133 and newer.

Replaying the resolver sequence on b10133 with these pins:

  OK  ggml-org ggml-org#24423    @ c3fb972
  OK  ggml-org ggml-org#24523    @ a58a7fa6e
  OK  unslothai #40      @ fee66f6a8

Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands.
danielhanchen pushed a commit to unslothai/llama.cpp that referenced this pull request Jul 26, 2026
ggml-org ggml-org#24523 now places LLM_ARCH_MINIMAX_M3 next to LLM_ARCH_MINIMAX_M2
instead of at the tail of the enum, matching how llama-arch.cpp already
groups the name table. That removes the src/llama-arch.h half of the
collision with ggml-org#25731, which appends LLM_ARCH_INKLING at the tail.

ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags
rename, so it now builds on its own against b10133 and newer.

Both head SHAs moved, so the pins and the carrier branch are refreshed.
unslothai #40 now only resolves what is left: both PRs still add a chat
parser and a detection block in the same region of common/chat.cpp,
immediately before "namespace workaround {". There is no principled
alternative anchor there, since chat.cpp has no MiniMax-M2 parser to sit
beside, so that half stays with the carrier.

Replaying the resolver sequence on b10133:

  OK  ggml-org ggml-org#24423    @ c3fb972
  OK  ggml-org ggml-org#24523    @ baee0f5
  OK  unslothai #40      @ 233cedb
danielhanchen added a commit to unslothai/llama.cpp that referenced this pull request Jul 26, 2026
#39 landed the first pin refresh only, so master still points at commits
that have since moved on and that conflict with each other.

ggml-org ggml-org#24523 was force-pushed: it now places LLM_ARCH_MINIMAX_M3 next
to LLM_ARCH_MINIMAX_M2 rather than at the tail of the enum, matching how
llama-arch.cpp already groups the name table. That removes the
src/llama-arch.h half of its collision with ggml-org#25731, which appends
LLM_ARCH_INKLING at the tail. New head baee0f5.

ggml-org ggml-org#25731 has picked up current master and the thinking_end_tags
rename, so it now builds on its own against b10133 and newer. Its entry
is replaced by unslothai #40, which carries what is still unresolvable
between the two: both add a chat parser and a detection block in the
same region of common/chat.cpp, immediately before "namespace
workaround {". chat.cpp has no MiniMax-M2 parser to sit beside, so
there is no principled alternative anchor for that half.

Replaying the resolver sequence on b10133:

  OK  ggml-org ggml-org#24423    @ c3fb972
  OK  ggml-org ggml-org#24523    @ baee0f5
  OK  unslothai #40      @ 233cedb

Drop the #40 entry and repin ggml-org#25731 upstream once either PR lands.

Co-authored-by: Daniel Han <unslothai@gmail.com>
@Firstbober

Copy link
Copy Markdown

What's the status of this PR? All checks are passing, and I don't see any code issues to be addressed and acted upon?

@nonetrix

Copy link
Copy Markdown

Thanks for still working on this even though it's been a while since weights dropped :D

# Conflicts:
#	common/arg.cpp
danielhanchen added a commit to unslothai/llama.cpp that referenced this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion CUDA Related to the CUDA backend examples ggml changes relating to the ggml tensor library for machine learning model Model specific Nvidia GPU Issues specific to Nvidia GPUs python python script changes testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.