Skip to content

Fix layer-wise (Muon) distributed optimizer + fp8-param - #5707

Closed
fanshiqing wants to merge 0 commit into
NVIDIA:mainfrom
fanshiqing:main
Closed

Fix layer-wise (Muon) distributed optimizer + fp8-param#5707
fanshiqing wants to merge 0 commit into
NVIDIA:mainfrom
fanshiqing:main

Conversation

@fanshiqing

@fanshiqing fanshiqing commented Jul 8, 2026

Copy link
Copy Markdown
Member
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Support Muon (LayerWiseDistributedOptimizer) with mxfp8 fp8-param-gather
fp8-param-gather relies on the DistributedOptimizer param-buffer machinery, which
LayerWiseDistributedOptimizer (Muon) does not have, so the optimizer's weight
update never reached the forward and loss stayed flat. Wire the missing pieces so
Muon trains with --fp8-param-gather + --reuse-grad-buf-for-mxfp8-param-ag at both
DP=1 and DP>1:

  • optimizer.py: the reuse_grad_buf (master -> fp8 param buffer) copy only applies
    to DistributedOptimizer; gate it on _copy_main_params_to_param_buffer so
    non-DistOpt optimizers (Muon's Float16 base opts) fall through to
    master -> model params. The forward then reads the update.

  • param_and_grad_buffer.py: in the LayerWise variable-size param all-gather, stage
    the high-precision fp32 master (param.main_param, cast to bf16) for MXFP8 params
    instead of flattening the MXFP8 wrapper (view(-1) is unsupported); the receive
    side re-quantizes via param.data.copy_. Also skip _post_param_sync for buckets
    with no param_data buffer (the LayerWise path already updated params in place).

@fanshiqing
fanshiqing requested review from a team as code owners July 8, 2026 10:04
@copy-pr-bot

copy-pr-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft July 8, 2026 10:04
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@fanshiqing
fanshiqing marked this pull request as ready for review July 8, 2026 10:05
@fanshiqing

Copy link
Copy Markdown
Member Author

/ok to test b9ebad6

@fanshiqing
fanshiqing requested a review from kvareddy July 8, 2026 10:06
@fanshiqing
fanshiqing requested a review from deepakn94 July 8, 2026 10:06
@fanshiqing
fanshiqing requested review from a team as code owners July 16, 2026 02:08
@fanshiqing
fanshiqing removed request for a team July 16, 2026 02:09
@fanshiqing fanshiqing changed the title Fix layer-wise (Muon) distributed optimizer crashes in common configs Fix layer-wise (Muon) distributed optimizer + fp8-param Jul 16, 2026
@fanshiqing

Copy link
Copy Markdown
Member Author

/ok to test eb367a4

@fanshiqing

Copy link
Copy Markdown
Member Author

Close this MR as this fix (fanshiqing@1b0121d) has already been merged with #4967 together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants