add DeepSeek-V4-Flash-0731 support and mxfp4->fp8 converter - #2717
Merged
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
yueming-yuan
changed the title
yueming-yuan
changed the title
Triton weight_dequant (and the sgl FP8 repackage convention) expect float32 blockwise scales; e8m0fnu KeyError'd inside triton during fp8_cast_bf16.
The broad except swallowed triton's KeyError('float8_e8m0fnu') from
weight_dequant and mislabeled it as a missing scale_inv tensor.
Official ue8m0 checkpoints (DeepSeek-V4-Flash-0731) store every fp8 scale as float8_e8m0fnu; cast to float32 (exact) before the triton dequant.
tp4 engines keep a per-rank memory-saver host shadow of the full engine weights (~75GB fp8 x 4 ranks/node); with the colocated Megatron actors loaded on top this OOMs the ~1TB host RAM. tp8 engines spanning 2 nodes halve the shadow.
…ry=True) Checkpoint loading churns through millions of MB-scale CPU tensors that glibc keeps in its arenas after free (below the 32MB dynamic mmap threshold), so actor RSS stays at the loading high-water mark (~26GB/rank observed). _host_emptyCache only covers the pinned cache; trim the glibc arenas too so the pre-sleep clear returns the pages before the offload backup is allocated.
yueming-yuan
requested review from
Shi-Dong,
Zhichenzzz,
fzyzcjy,
guapisolo,
jybsuper and
maocheng23
as code owners
August 23, 2026 20:06
Frozen params (requires_grad=False, e.g. --moe-router-freeze-gate) are excluded from the DDP param buffers so they have no master weight to rebuild from; back them up alongside the fp32 params. DSV4 hit this via its frozen router gate.
The loading residency it targeted is not glibc free arena pages (trim reclaimed ~0.05GB against a ~26-38GB residual); root cause still open.
yushengsu-thu
approved these changes
Aug 24, 2026
guapisolo
added a commit
that referenced
this pull request
Aug 24, 2026
PR #2717 treated every four-GPU node as the GB300 full-model profile, which routed the single-node 4-layer H200 smoke test into TP8/EP8. Exclude the 4-layer model from the GB300 branch and keep only the two launcher regression cases that distinguish the intended profiles.
|
Hi @yueming-yuan - did you test what logprob differences ended up being between rollout/actor by chance? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
DeepSeek-V4-Flash-0731(official deepseek-ai release, MXFP4 routed experts) to the DeepSeek-V4 launcher. The architecture is identical to DeepSeek-V4-Flash, so this is wiring plus one new conversion stage:tools/convert_mxfp4_to_fp8.py: casts packed-e2m1fn expert weights (int8 + per-(1,32)-block ue8m0 scales) losslessly to e4m3fn with (128,128)-block e8m0 scales — the samecast_e2m1fn_to_e4m3fnthe officialinference/convert.pyuses, producing the sgl-project FP8 repackage layout. All other tensors are copied unchanged;expert_dtypeis dropped from config.json.scripts/run_deepseek_v4.py: new--model-name DeepSeek-V4-Flash-0731(org deepseek-ai, megatron typedeepseek-v4-flash), newprepare-fp8stage chained intofull-train(sentinel-skipped), rollout/bf16 stages read from the cast FP8 dir. Downstream (fp8_cast_bf16, torch_dist conversion, fp8 rollout) is byte-identical to the DeepSeek-V4-Flash-FP8 path.Validation
tests/fast/launch_scripts: 39 passed;tests/manual/launch_scripts: 455 passed (the 4scripts/amd/*errors are pre-existing on main),prepare_fp8snapshot recorded.