Skip to content

cp: fix: use local_rank (#2328)#2329

Closed
ko3n1g wants to merge 34 commits intomainfrom
r0.3.0
Closed

cp: fix: use local_rank (#2328)#2329
ko3n1g wants to merge 34 commits intomainfrom
r0.3.0

Conversation

@ko3n1g
Copy link
Copy Markdown
Contributor

@ko3n1g ko3n1g commented Feb 11, 2026 •
edited by coderabbitai bot
Loading

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Changelog

  • Add specific line by line info of high level changes in this PR.

GitHub Actions CI

See the CI sectionin the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • Related to # (issue)

Summary by CodeRabbit

  • New Features

    • Added Multi-Token Prediction documentation with configuration guidance and examples.
    • Added Ministral3 Vision-Language Model support with examples and recipes.
    • Added GLM-4.5V examples with conversion, inference, and finetuning scripts.
    • Introduced packed sequence support for vision-language model training with validation.
    • Added PEFT (LoRA/DoRA) finetuning for Qwen3-VL models.

  • Bug Fixes

    • Fixed VLM forward pass compatibility for multiple return types.
    • Fixed Ministral3 image feature extraction to use pooler output.
    • Fixed inference wrapper decoder exposure for Qwen models.
    • Addressed CVE-2025-68973 in Docker image.

  • Documentation

    • Updated release version to 0.3.0.
    • Enhanced GLM-4.5V and Qwen3-VL documentation with PEFT examples.
    • Updated documentation links and references for vision-language models.

  • Tests

    • Added packed sequence finetuning tests for multiple models.
    • Added Qwen3-VL finetuning test suite.
    • Added validation tests for packed sequence configurations.

  • Chores

    • Updated dependencies and GitHub workflows.
    • Updated Megatron-LM submodule.
    • Adjusted parallelism configurations for performance tuning.
    • Enhanced shell scripts for model workflows.

ko3n1g and others added 30 commits February 2, 2026 15:52
@ko3n1g
chore: Update docs (#2178)
1e63871 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g @dingqingy-nv
cp: Dsv3 Recipe Update (2152) into r0.3.0 (#2186)
7f61c05 
Signed-off-by: Dingqing Yang <dingqingy@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Dingqing Yang <dingqingy@nvidia.com>
@ko3n1g @cuichenx
cp: Revert packed seq extra checks (2180) into r0.3.0 (#2196)
bf5ee44 
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
@ko3n1g @malay-nagda
cp: `DSv3 EP=8 for B200, PP8-VP2 for B300 BF16, Lm3.1 405B TP4-CP1 GB…
48a27fa 
…300 FP8-CS (2175)` into `r0.3.0` (#2198)

Signed-off-by: Malay Nagda <malayn@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: malay-nagda <malayn@nvidia.com>
@ko3n1g @cuichenx @coderabbitai
cp: [docs] add MTP guide (2138) into r0.3.0 (#2202)
77f5503 
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <cxcui@alumni.cmu.edu>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
@ko3n1g @yashaswikarnati
cp: add peft to recipe qwen3vl (2023) into r0.3.0 (#2220)
c22f858 
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com>
@ko3n1g @yaoyu-33
@suiyoubi @ananthsub @github-actions @athitten @dingqingy-nv @cuichenx @malay-nagda @chtruong814
cp: `[doc, model] feat: Add GLM-4.5V VL examples and update Gemma 3 V…
3cb4dee 
…L docs (2151)` into `r0.3.0` (#2226)

Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: Ao Tang <aot@nvidia.com>
Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>
Signed-off-by: Dingqing Yang <dingqingy@nvidia.com>
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Malay Nagda <malayn@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>
Co-authored-by: Ao Tang <aot@nvidia.com>
Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: Dingqing Yang <dingqingy@nvidia.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Chen Cui <chcui@nvidia.com>
Co-authored-by: malay-nagda <malayn@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@ko3n1g @kamran-nvidia
cp: [docs, model] Add Ministral 3 Examples (2139) into r0.3.0 (#2204
f91a086 
)

Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: kamran-nvidia <kjafarisadeg@nvidia.com>
@ko3n1g
cp: ci(fix): Wheel build (2192) into r0.3.0 (#2238)
94af2ed 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
@ko3n1g
cp: chore: Expose custom bash cmds (2237) into r0.3.0 (#2243)
98762f1 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
@ko3n1g @meatybobby
cp: Fix Qwen2.5-VL huggingface conversion issue (#2107) (2156) into…
37ba134 
… `r0.3.0` (#2205)

Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: meatybobby <meatybobby@gmail.com>
@ko3n1g
cp: fix: Use nargs for custom_bash_cmds (2261) into r0.3.0 (#2262)
ae58d30 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
@ko3n1g @malay-nagda
cp: gb300 lm3.1 495b nvfp4 fix (2258) into r0.3.0 (#2259)
b6661ea 
Signed-off-by: Malay Nagda <malayn@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: malay-nagda <malayn@nvidia.com>
@ko3n1g @youngeunkwon0405
cp: Fix: perf script ddp nccl-ub (2158) into r0.3.0 (#2217)
241572b 
Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>
@ko3n1g @dingqingy-nv
cp: `Update Qwen3 235B A22B MXFP8 GB200/300 recipe and resolve NaN gr…
d7a13b1 
…ad norm (2209)` into `r0.3.0` (#2210)

Signed-off-by: Dingqing Yang <dingqingy@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Dingqing Yang <dingqingy@nvidia.com>
@ko3n1g @malay-nagda
cp: b300 dsv3 bf16 hang fix (2260) into r0.3.0 (#2270)
78a5eba 
Signed-off-by: Malay Nagda <malayn@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: malay-nagda <malayn@nvidia.com>
@ko3n1g @yaoyu-33
chore: Change submodule pointer for release (#2191)
98506a7 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>
@ko3n1g
cp: feat: Add dataset compile helper (#2236) (#2249)
8ae972e 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Revert "cp: Dsv3 Recipe Update (2152) into r0.3.0 (#2186)"
843c2d7 
This reverts commit 7f61c05.
@ko3n1g
fix no submodule checkout
6d665f8 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
Revert "cp: `Update Qwen3 235B A22B MXFP8 GB200/300 recipe and resolv…
34aec47 
…e NaN grad norm (2209)` into `r0.3.0` (#2210)"

This reverts commit d7a13b1.
@ko3n1g
Merge pull request #2271 from NVIDIA-NeMo/ko3n1g/fix/r030
861bbdd 
Revert #2152 and 2209
@ko3n1g
Reapply "cp: `Update Qwen3 235B A22B MXFP8 GB200/300 recipe and resol…
f2fee27 
…ve NaN grad norm (2209)` into `r0.3.0` (#2210)"

This reverts commit 34aec47.
@ko3n1g
Reapply "cp: Dsv3 Recipe Update (2152) into r0.3.0 (#2186)"
595e767 
This reverts commit 843c2d7.
@ko3n1g
Merge pull request #2273 from NVIDIA-NeMo/ko3n1g/chore/reapply-2152-a…
a7a840d 
…nd-2209

Ko3n1g/chore/reapply 2152 and 2209
@ko3n1g @malay-nagda
cp: dsv3_gb300_revert- BF16 & FP8-MX scale (#2277) (#2286)
1db8398 
Signed-off-by: Malay Nagda <malayn@nvidia.com>
Co-authored-by: malay-nagda <malayn@nvidia.com>
@ko3n1g
cp: mlflow upgrade (#2281)
b39bd94 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
cp: build: Address CVE-2025-68973 (#2290)
be11e50 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g @ananthsub
cp: `docs: Update callback code snippets to include all imports neede…
b10d7e3 
…d for example (2283)` into `r0.3.0` (#2291)

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ko3n1g
cp: build: Bump modelopt and TE (2304) into r0.3.0 (#2314)
669ad62 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malay-nagda malay-nagda Awaiting requested review from malay-nagda malay-nagda is a code owner

@erhoo82 erhoo82 Awaiting requested review from erhoo82 erhoo82 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ko3n1g