Skip to content

metal: deepseek v4 optimize token generation - #1

Closed
tarruda wants to merge 2 commits into
dsv4-metal-sparse-attentionfrom
dsv4-metal-tokengen
Closed

metal: deepseek v4 optimize token generation#1
tarruda wants to merge 2 commits into
dsv4-metal-sparse-attentionfrom
dsv4-metal-tokengen

Conversation

@tarruda

@tarruda tarruda commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Overview

Picking a couple of commits from my branch which improve DSv4 token generation

Additional information

This builds on ggml-org#26512 and will improve token generation and keep it more stable as context grows. Here's -d 0,10000,20000,30000 -n 128 -p 2048 -b 2048 -ub 2048 on a M1 Ultra:

| model                          |       size |     params | backend    | threads | n_ubatch |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -------: | --: | --------------: | -------------------: |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 |          pp2048 |        199.61 ± 0.96 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 |           tg128 |         25.74 ± 0.03 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 | pp2048 @ d10000 |        190.85 ± 0.65 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 |  tg128 @ d10000 |         22.82 ± 0.02 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 | pp2048 @ d20000 |        182.61 ± 0.75 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 |  tg128 @ d20000 |         22.37 ± 0.04 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 | pp2048 @ d30000 |        175.11 ± 0.27 |
| deepseek4 ?B Q8_0              |  91.94 GiB |   284.33 B | MTL,BLAS   |      16 |     2048 |   1 |  tg128 @ d30000 |         22.09 ± 0.08 |

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yes, 100% GPT 5.6 Sol. I'm not a ML kernel programmer and have no idea what I'm doing. If anyone wants to create a new PR that cleans up, improves or rewrite these changes, they have my blessing to do so.

tarruda added 2 commits August 3, 2026 09:51
Reduce DeepSeek V4 decode overhead with fused compressor and mask operations,
avoid redundant Lightning Indexer work, and bound long-context attention by
gathering only selected compressed keys.

- Add DSV4_COMPRESS and DSV4_TOP_K_MASK with CPU references and capability
  probes.
- Skip Lightning Indexer and TOP_K when every compressed row is selected.
- Reuse sparse packing for deep single-token decode with dense fallbacks.
- Test fused compression, mask construction, and compact Flash Attention.

Assisted-by: Codex
Fuse compressor and attention-mask construction on Metal and keep
long-context decode bounded by gathering only selected compressed keys.

- Add kernels for DSV4_COMPRESS and DSV4_TOP_K_MASK.
- Skip K and V loads for fully masked vector Flash Attention groups.
- Extend DSV4_SPARSE_PACK with a parallel compressed-only decode mode.
- Preserve compact F16 masks without intermediate conversions.

Assisted-by: Codex
@tarruda tarruda closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant