Add DeepSeek-V4 model and safetensors/tokenizer fallback handling

[austinbv]

Adds deepseek_v4.py implementing DeepSeek-V4-Flash architecture: MLA attention with inverse RoPE, hash-based expert routing (HC-Sinkhorn), MXFP8-quantized MoE layers, and sliding-window/full-attention interleaving.

Adds _load_weights_file() to handle safetensors shards that contain F8_E8M0 block-scale tensors (which mx.load rejects). The fallback parser decodes F8_E8M0 as unsigned exponents (2^(b-127)), BF16 via uint16 view, and maps F8_E4M3/F8_E5M2 to uint8 to satisfy mx.from_fp8's documented input contract.

Wraps AutoTokenizer.from_pretrained in a try/except so models whose model_type is unknown to transformers fall back to PreTrainedTokenizerFast.

[Blaizzy]

Hey @austinbv

I would recommend you check Deepseek v3 for the FP8 quantisation part. It can be done in the sanitize method by dequantizing the weights at load time and added or targeted to the first DeepSeek v4 PR #1189 alongside the missing tests (test_models.py).

Regarding the tokenizer, I wouldn't patch it at all, because it's a matter of waiting for this PR to be merged into transformers. So a simple note about it would suffice.

huggingface/transformers#45616

[trevorgordon981]

Ran this PR against mlx-community/deepseek-ai-DeepSeek-V4-Flash-8bit on an M3 Ultra. Loading and in-process generation both work; the HTTP server path doesn't.

What works

mlx_lm.load() accepts the repo's weight key layout cleanly (thanks to _load_weights_file() unpacking F8_E8M0/E4M3/E5M2 blocks):

load(): 24.3s, peak_rss=281.9 GB

mlx_lm.generate() in-process produces a coherent first answer:

>>> generate(model, tok, prompt="The capital of France is", max_tokens=50)
"Paris.\nThe capital of France is Paris.\nThe capital of France is Paris..."

(It loops after the correct first answer, but that's just the default sampler with no repetition penalty. The first token is right and the model is clearly running.)

What doesn't

Starting python -m mlx_lm.server --model mlx-community/deepseek-ai-DeepSeek-V4-Flash-8bit --port 18082 and hitting /v1/completions with the same prompt returns garbage tokens:

python -m mlx_lm.server --model mlx-community/deepseek-ai-DeepSeek-V4-Flash-8bit \
    --host 127.0.0.1 --port 18082 &
# wait for "Starting httpd at..."

curl -s -X POST http://127.0.0.1:18082/v1/completions \
    -H 'Content-Type: application/json' \
    -d '{"model": "mlx-community/deepseek-ai-DeepSeek-V4-Flash-8bit",
         "prompt": "The capital of France is",
         "max_tokens": 15}'
# → choices[0].text = "otold  ,   ,        ( ("

/v1/chat/completions with "Say hello in one short sentence" returns equally bad output:

" A   A  P  P  P  P P P P P P P P P P P P  U  P  P"

The chat-template path and the raw /v1/completions path fail the same way, so it's not a template issue. It's specific to the server's streaming/detokenizer path. In-process generate() is fine with the same model instance and identical sampler defaults.

My guess (haven't confirmed with a patch): the server's token-by-token detokenizer is emitting individual bytes for deepseek_v4's tokenizer output rather than waiting for complete surface-form pieces. The tokenizer-fallback handling in this PR's tokenizer_utils.py only covers the AutoTokenizer.from_pretrained case, not the streaming detok path used by the server.

Suggested next step

A server smoke test that runs /v1/completions after mlx_lm.load() validates would catch this. I can submit one as a follow-up PR.

Side observation

/v1/models returns every model in ~/.cache/huggingface/hub, not just the one the server was started with. On a machine with multiple MLX models cached this makes the response confusing (mine lists Qwen/Qwen3-Embedding-8B first even though the server is running V4-Flash). I haven't verified whether this behavior is from this PR or pre-existing in mlx_lm.server.

Scoreboard for reference

I ran the three open V4 PRs through the same harness on the same machine + the same repo:

• feat: add DeepSeek-V4 (Pro/Flash) model support #1189: ❌ ValueError: Received 1423 parameters not in model (weight keys don't match)
• Add DeepSeek-V4 model and safetensors/tokenizer fallback handling #1190 (this PR): ✅ load, ✅ generate, ❌ server
• Add DeepSeek-v4 (Flash/Pro) #1192: ❌ same ValueError as feat: add DeepSeek-V4 (Pro/Flash) model support #1189

So this PR is currently the only one that loads the public weights at all. That's a big deal worth landing, but the server bug should probably be resolved first since mlx_lm.server is how most people will actually run it.

[austinbv]

Closing as there are better options