Add DeepSeek-v4 (Flash/Pro) - #1192

Closed

Blaizzy wants to merge 62 commits into

ml-explore:mainfrom

Blaizzy:pc/add-deepseekv4flash-model

Commits on Nov 12, 2025

fix input_embeddings prefill bug in generate_step
Blaizzy
committed
2bfc361
format
Blaizzy
committed
5cf134c

Commits on Dec 2, 2025

Merge branch 'ml-explore:main' into main
Blaizzy
authored
2e09bd2

Commits on Mar 4, 2026

Merge branch 'ml-explore:main' into main
Blaizzy
authored
d59907c

Commits on Apr 2, 2026

Merge branch 'ml-explore:main' into main
Blaizzy
authored
e06ca01

Commits on Apr 24, 2026

Merge branch 'ml-explore:main' into main
Blaizzy
authored
5c965a5
add DS4
Blaizzy
committed
18c12cb
Fix DeepSeek V4 Flash generation math
Blaizzy
committed
28d991f
Generalize safetensors dtype fallback
Blaizzy
committed
be714fd
Remove deprecated DeepSeek V4 tests for sanitization and model loading
Blaizzy
committed
a833f26
Refactor DeepSeek V4 tests to consolidate and clarify test cases
Blaizzy
committed
28a83ea
format
Blaizzy
committed
bc1c285
keep experts quantized
Blaizzy
committed
f4f7b4d
fix fp32 promotion (3 tps -> 11 tps)
Blaizzy
committed
e3221c4
Fix DeepSeek V4 batched routing
Blaizzy
committed
16b268a
compile _limited_swiglu, hc_split_sinkhorn and make robust predicate
Blaizzy
committed
201267d
Enhance quant_predicate to support new projection modes and add @mx.compile decorator to _limited_swiglu function
Blaizzy
committed
f26e519
keep experts in mxfp4
Blaizzy
committed
61e006e
Refactor quantized linear layer
Blaizzy
committed
f2b4e9d
Implement optimized Sinkhorn operations in DeepSeek V4 and add corresponding unit tests for HyperConnection
Blaizzy
committed
c0d9222
Optimize matrix multiplication in HyperConnection by replacing einsum with matmul for improved performance
Blaizzy
committed
ef8c95d
HC sinkhorn normalization
Blaizzy
committed
f7ff216
fix(deepseek_v4): numerical stability and compressed attention mask padding

eauchs
and
machiabeli
committed
c6a7828
Merge pull request #14 from eauchs/fix-blaizzy-pr1192
Blaizzy
authored
f8ebcaf

Commits on Apr 25, 2026

Refactor DeepSeek V4 scoring and compile rope full
Blaizzy
committed
2a73e76
Add DeepseekV4SwitchGLU and BatchRotatingKVCache; refactor position handling in DeepseekV4RoPE and DeepseekV4Cache
Blaizzy
committed
c2c9801
Refactor DeepseekV4Cache to improve state management and add length handling for batching
Blaizzy
committed
8ac37c0
format
Blaizzy
committed
369fd37
Revert "keep experts in mxfp4"
Blaizzy
committed
166bba6
Revert quant predicate (lower precision experts)
Blaizzy
committed
cbb0b72
Remove quantization_config from ModelArgs in deepseek_v4.py
Blaizzy
committed
e3313d4
Update deepseek_v4.py

Blaizzy
and
kernelpool
authored
16de966

Commits on Apr 26, 2026

Fix RoPE init
Blaizzy
committed
01bcd7d
Refactor RoPE initialization in V4Attention to streamline handling of compressed layers and update associated tests for consistency.
Blaizzy
committed
8bd10c6
Improve DeepseekV4Cache and V4Attention with improved trimming logic and mask handling for compressed layers
Blaizzy
committed
014a10b
Add scores parameter and updatie projection logic
Blaizzy
committed
5d1a67a
Optimize DeepSeek V4 performance

0xClandestine
and
Blaizzy
committed
26f49f5
Refactor dtype handling in _ensure_cached method and improve pooled tensor processing logic (fix cache blowing up)
Blaizzy
committed
2ba4215
format
Blaizzy
committed
16f7205
Fix matrix multiplication in _hc_expand_op to use original comb tensor instead of swapped axes
Blaizzy
committed
193aa77
Refactor output projection
Blaizzy
committed
5a4aaa4
Refactor kv tensor reshaping in V4Attention for improved dimensionality handling
Blaizzy
committed
2591b51
Remove redundant cache type check in V4Attention to streamline processing logic
Blaizzy
committed
22da01a
Fix DeepSeek V4 sparse pooled prefill memory

Blaizzy
and
adurham
committed
8e8571a
Remove sparse pooled attention test block
Blaizzy
committed
ddeffe3

Commits on Apr 27, 2026

Commits on Apr 28, 2026

Enable loading the original checkpoint
angeloskath
committed
f83460f
Simplify GLU and gate remove intermediate castings
angeloskath
committed
81a8c57
Fix RoPE to use the kernel by scaling freqs
angeloskath
committed
4951496

Commits on Apr 29, 2026

Start simplifying and speeding up the attention
angeloskath
committed
3cf5282
Attempt to fix IndexError on batching.
pcuenca
committed
ad9e329
Refactor compressor and compressed non-sparse attention
angeloskath
committed
83d7e74
Ensure dtype going into mx.fast.rope is int32
pcuenca
committed
d8a47ae
Merge pull request #21 from pcuenca/rope-dtype
Blaizzy
authored
42a416b
Merge pull request #20 from pcuenca/index-error
Blaizzy
authored
85cb244

Commits on May 1, 2026

Major cache refactor and attention simplification
angeloskath
committed
2120a1b
Simplify HyperConnection
angeloskath
committed
ca8b299
Add hyper_connection.py
angeloskath
committed
c58834b
Fix batch cache edge case
angeloskath
committed
7efb57f
Fix tensor parallel distributed
angeloskath
committed
5c10538