Repository navigation
Conversation
DeepSeek-V4.1-Flash-MXFP4 is a mixed-precision Quark export: MXFP4 for the experts globally, plus 226 per-layer specs putting attention on 32x32 per-block FP8 with e8m0 scales. Four things stop it from loading, each of which assumes DeepSeek V4's 128x128 geometry or the native scale naming. DeepseekV4FP8Config.override_quantization_method claimed any Quark export whose global scheme is MXFP4, but from_config rewrites the config into a single global FP8 scheme, discarding the per-layer specs. Decline exports that carry layer_quant_config so QuarkConfig handles them. QuarkConfig then had no scheme for those layers: _is_fp8_w8a8 only accepts 128x128 per-block FP8 and the MX matchers only accept 1-D per-group weights, so dispatch raised NotImplementedError. A 32x32 per-block e8m0 weight is MXFP8 in a 2-D wrapper, which ModelOpt's MXFP8 linear method already reads, so route it there. The VL wrapper exposed neither packed_modules_mapping nor hf_to_vllm_mapper on the class. configure_quant_config reads both off the class before __init__ runs, so per-layer keys naming layers.N.attn.wq_a never matched the runtime language_model.model.layers.N.attn.wq_a and silently fell back to the global MXFP4 spec. Finally, Quark spells the engram scale embed.weight_scale; the mapper's \.scale$ rules only match a literal ".scale" suffix, so the tensor was never routed. Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
Route Quark's 32x32 per-block e8m0 FP8 weights through a Quark-native QuarkW8A8Mxfp8 scheme rather than borrowing ModelOptLinearMethod, so the ROCm/Quark path no longer reaches into the ModelOpt namespace for a format Quark itself exports. The scheme wraps the same shared init_mxfp8_linear_kernel() used by the compressed-tensors and INC MXFP8 schemes. Quark spells the scale 2-D (one e8m0 value per block_size[0] weight rows) where MXFP8 canonically carries one per row, so the scale loader expands each row on load. QuarkOCP_MX is deliberately not extended: it covers the sub-byte packed MX formats and has no MXFP8 entry in _WEIGHT_QUANT_KEY_MAP, whereas MXFP8 is unpacked. Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
Make MXFP8 a first-class OCP MX dtype so it routes through the normal Quark matcher chain like MXFP4 and MXFP6, eliminating the short-circuit (`_mxfp8_block_linear_method`) that caused `get_quant_method` and `get_quant_method_target` to diverge on MXFP8 layers. Both MXFP8 config dialects are now accepted: canonical per-group (qscheme=per_group, group_size=32, scale_format=e8m0) and V4.1's 2-D per-block (qscheme=per_block, block_size=[R,32], scale_type=float8_e8m0fnu). Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com> Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
xiao-llm
requested review from
AndreasKaratzas,
dllehr-amd,
hongxiayang,
mgoin,
pavanimajety,
robertgshaw2-redhat,
tjtanaa,
tlrmchlsmth,
yewentao256 and
zyongye
as code owners
September 16, 2026 14:46
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mergify
Bot
added
deepseek
xiao-llm
changed the title
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix
DeepSeek-V4.1-Flash-MXFP4loading issue. It is a mixed-precision AMD-Quark export — MXFP4 experts, but 226 per-layer attention specs in MXFP8:global_quant_configfp4/per_group/group_size=32/e8m0layer_quant_config(226 entries)fp8_e4m3/per_block/block_size=[32,32]/float8_e8m0fnuExsiting DeepSeek V4's require
128x128block geometry, or the DeepSeek-native scale naming.This PR fix the blockers and support MXFP8 by routing through the normal Quark matcher chain via QuarkOCP_MX, like MXFP4 and MXFP6. Changes (8 files, +300/−241):
End-to-end
DeepSeek-V4.1-Flash-MXFP4(48 shards, 475 GB) on 4x MI355X at TP=4: allshards load in 74.5 s, MXFP8 attention resolves to
QuarkW8A8Mxfp8, and themodel is coherent.
Accuracy was measured rather than eyeballed —
lm_evalgsm8k, full 1319problems, 5-shot, greedy
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.