Glimmer drafter optimization

[pcuenca]

As discussed offline with @ruixiang63 @ngxson @CISC @seyeong-han

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes

[pcuenca]

As explained by @ruixiang63:

This change is not ideal. At the moment, llama.cpp does not support GPU backend sampling with multiple tokens. There is an upcoming PR that aims to address this: #25532
I’d suggest waiting for that PR to be merged, then revisiting this change so we can implement proper multi-sequence, multi-token GPU sampling.

[pich]

@pcuenca For reference, on my RTX PRO 4000 Blackwell 24 GB with Glimmer 30B Q5_K_M:

Regular decoding: 17.98 tok/s
DFlash before this optimization: ~38 tok/s
DFlash with this PR: 84.64 tok/s

So this optimization gave me roughly 2.2× higher DFlash throughput, and 4.7× vs regular decoding.

My bench: https://piszczek.pl/blog/dflash-changes-what-tokens-per-second-means

[ruixiang63]

I am working on a PR to implement simpler and more robust GPU backend sampling for DFlash.

[ruixiang63]

Please check this PR: #26958

[pich]

@ruixiang63 tested #26958 against the specialized #26842 path on my Glimmer setup.

Same machine and same config:

RTX PRO 4000 Blackwell SFF 24 GB, 70 W
Debian 13.6, kernel 6.12.38
driver 610.57.04, CUDA 12.9
Muse Glimmer 30B Q5_K_M, 256K ctx, Q8 KV, DFlash F16 KV, n_max=15, mmproj loaded

10 runs, 1024 tokens each:

#26842: 80.05 tok/s avg
#26958: 80.69 tok/s avg

So basically parity, with #26958 slightly faster in my test.

Acceptance was exactly the same: 38.67%, mean accepted length 6.77.

CPU was also the same, around one core.

The only noticeable difference I saw was VRAM:

#26842: 22,852 MiB
#26958: 23,052 MiB

So around +200 MiB for the generic backend sampling path.

I also reran my mixed workload and got 37.51 tok/s / 14.41% acceptance, almost identical to the previous result, so that part still looks workload/predictability dependent rather than implementation dependent.

Full 262,116 token context fill also passed, no OOM, peak VRAM 23,120 / 24,467 MiB.

For me #26958 looks like the better implementation direction. Same performance as the dedicated DFlash argmax path, but cleaner and reusable for DFlash/DSpark.

One thing I would probably look at next is the extra VRAM. For greedy top_k(1) maybe the backend only needs sampled IDs, without keeping full logits/probs/candidate buffers for every output row. That might recover most of the 200 megz

[pcuenca]

Superseded by #26958