Skip to content

Add shared outer lora - #3958

Merged
yaoyu-33 merged 8 commits into
NVIDIA-NeMo:mainfrom
nanjiangwill:shared-outer-lora
Jun 10, 2026
Merged

Add shared outer lora#3958
yaoyu-33 merged 8 commits into
NVIDIA-NeMo:mainfrom
nanjiangwill:shared-outer-lora

Conversation

@nanjiangwill

@nanjiangwill nanjiangwill commented May 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do ?

follow the shared outer lora design here sgl-project/sglang#21466.

e2e from radixark#3

Changelog

  • Add specific line by line info of high level changes in this PR.

GitHub Actions CI

See the CI section in the Contributing doc for how to trigger the CI. A Nvidia developer will need to approve and trigger the CI for external contributors.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • Related to # (issue)

@copy-pr-bot

copy-pr-bot Bot commented May 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:peft Parameter-efficient fine-tuning (LoRA, adapters) feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge waiting-on-customer Waiting on the original author to respond labels May 23, 2026
@nanjiangwill nanjiangwill mentioned this pull request May 26, 2026
Open
yaoyu-33
yaoyu-33 reviewed May 26, 2026
View reviewed changes
Comment thread src/megatron/bridge/models/conversion/peft_bridge.py
yaoyu-33
yaoyu-33 reviewed May 26, 2026
View reviewed changes
Comment thread src/megatron/bridge/models/conversion/peft_bridge.py
@yaoyu-33

yaoyu-33 commented May 26, 2026

Copy link
Copy Markdown
Contributor

/claude review

claude[bot]
claude Bot reviewed May 26, 2026
View reviewed changes
Comment thread src/megatron/bridge/peft/utils.py
claude[bot]
claude Bot reviewed May 26, 2026
View reviewed changes
Comment thread src/megatron/bridge/peft/utils.py Outdated
@claude

claude Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

Light Code Review - Critical: AssertionError typo (utils.py:1676) - except AssertionError will never match, Python has no built-in AssertionError, at runtime this raises NameError instead of gracefully returning, must be AssertionError, see inline comment. Minor: Missing trailing newline at end of utils.py, see inline comment. Test coverage: This PR adds three new classes but no tests are included. The existing GroupedExpertLinearAdapter has extensive unit tests, the new adapter should have comparable coverage. Suggested test cases: SharedOuterGroupedExpertAdapter forward pass fc1 and fc2 paths, sharded_state_dict for both, PackedPerExpertLinear forward and sharded_state_dict, _make_cross_ep_replicated broadcast and gradient hook, LoRA.transform with experts_shared_outer_loras=True, stream_adapter_weights_megatron_to_hf shared-outer branch for fc1 and fc2, shared-outer fused fc1 linear_out slicing. No perf tests impacted.

yaoyu-33
yaoyu-33 previously approved these changes May 27, 2026
View reviewed changes
@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and only waiting for CI to pass before merge and removed waiting-on-customer Waiting on the original author to respond labels May 27, 2026
@yaoyu-33

yaoyu-33 commented May 27, 2026

Copy link
Copy Markdown
Contributor

/ok to test 46663cf

@yaoyu-33

yaoyu-33 commented May 28, 2026

Copy link
Copy Markdown
Contributor

/ok to test eaa9592

@yaoyu-33 yaoyu-33 removed needs-review PR is ready for code review and waiting on a reviewer waiting-on-customer Waiting on the original author to respond labels Jun 1, 2026
yaoyu-33
yaoyu-33 previously approved these changes Jun 6, 2026
View reviewed changes
@yaoyu-33 yaoyu-33 added the ready-to-merge PR is approved, current, and only waiting for CI to pass before merge label Jun 6, 2026
@yaoyu-33

yaoyu-33 commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

/ok to test f789f92

@yaoyu-33
yaoyu-33 enabled auto-merge (squash) June 6, 2026 04:14
@yaoyu-33

yaoyu-33 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 8e3a2eb

@yaoyu-33
fix(peft): import parallel_state in utils
5fbd49d 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

yaoyu-33 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 5fbd49d

@adityavavreNVDA
Merge branch 'main' into shared-outer-lora
20136f5 
Signed-off-by: adityavavreNVDA <avavre@nvidia.com>
@adityavavreNVDA

adityavavreNVDA commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

/ok to test 20136f5

@yaoyu-33
yaoyu-33 merged commit fbf2d3c into NVIDIA-NeMo:main Jun 10, 2026
21 of 25 checks passed
vasunvidia pushed a commit to vasunvidia/Megatron-Bridge that referenced this pull request Jun 10, 2026
@nanjiangwill @vasunvidia
Add shared outer lora (NVIDIA-NeMo#3958)
ccff5cb 
Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
@cuichenx cuichenx mentioned this pull request Jul 28, 2026
Open
@HollowMan6 HollowMan6 mentioned this pull request Aug 14, 2026
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@yaoyu-33 yaoyu-33 yaoyu-33 left review comments

Assignees

No one assigned

Labels

area:peft Parameter-efficient fine-tuning (LoRA, adapters) community-request feature New capabilities, enhancements, or enablement work needs-more-tests Requires additional L0 and L1 test coverage before merge ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nanjiangwill @yaoyu-33 @JRD971000 @adityavavreNVDA @svcnvidia-nemo-ci