Skip to content

feat: AUT-2177 add Ling-V3 Tiny hybrid training support - #6974

Closed
svcnemo-autobot wants to merge 36 commits into
NVIDIA:mainfrom
svcnemo-autobot:chore/cherrypick-14e28c6e-to-main-v3
Closed

feat: AUT-2177 add Ling-V3 Tiny hybrid training support#6974
svcnemo-autobot wants to merge 36 commits into
NVIDIA:mainfrom
svcnemo-autobot:chore/cherrypick-14e28c6e-to-main-v3

Conversation

@svcnemo-autobot

@svcnemo-autobot svcnemo-autobot commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Background

Port the Ling-V3 Tiny hybrid-model training support from PR #6556 to main. This replacement preserves the corrected head after the prior PR branch became unavailable to this workflow.

What changed

  • Add Ling-V3 Tiny hybrid model and KDA/GDN training support.
  • Adapt imports and integrations to the main source layout.
  • Secure recipe credential cleanup and legacy checkpoint loading.
  • Apply the Black formatting required by the prior CI run.

Details

flowchart LR
  A[Ling-V3 configuration] --> B[Hybrid model]
  B --> C[KDA and GDN training]
Loading

Tested

  • Source-layout and conflict-marker inspection passed.
  • Confirmed the exact Black fix is present in gdn.py and kda.py.
  • DCO passes on 91f6f85a.
  • Full autoformat could not run because the typed checkout exposes no .git metadata; GitHub CI remains authoritative.

fkuner and others added 30 commits August 27, 2026 21:45
@fkuner @svcnemo-autobot
Prepare the Gated DeltaNet base for channel-wise variants
139b583 
Keep the existing private GDN base as the shared implementation for head-wise GatedDeltaNet and channel-wise variants. Generalize its variant hooks, parameter dtype handling, diagnostics, and norm-output recompute contract without exposing a broader linear-attention API.

Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 6ff2fcc)
@fkuner @svcnemo-autobot
Add Kimi Delta Attention to hybrid models
da9d990 
Add KDA as the channel-wise Gated DeltaNet variant in HybridModel. Reuse the existing private GDN-family base and gdn_norm_out recompute selector while keeping KDA projection, kernel, and context-parallel behavior variant-specific.

Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 16f448c)
@fkuner @svcnemo-autobot
Honor GDN projection metadata in FSDP checkpoints
b382bfe 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 72cefc3)
@fkuner @svcnemo-autobot
Add head-wise output gating to MLA
1f34327 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit ed853d3)
@fkuner @svcnemo-autobot
Include nested MTP modules in QK clipping
dc1ca15 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit fe92b2d)
@fkuner @svcnemo-autobot
Add a Ling-V3 Tiny hybrid model spec
98d52cf 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 2ec2297)
@fkuner @svcnemo-autobot
Add a Ling-V3 Tiny pretraining example
f4cafaf 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 5fe61a8)
@fkuner @svcnemo-autobot
Account for KDA and MLA in training metrics
94b5ca0 
Count KDA and MLA operations in the hybrid training FLOPs estimate so that Ling-V3 Tiny TFLOPs and MFU include every attention layer used by the model.

Keep the architecture-specific formulas separate and cover direct KDA projections and state updates, MLA latent projections, output gating, and ragged dense attention.

Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 8296e8b)
@fkuner @svcnemo-autobot
Fix Hybrid MTP z-loss metric indexing
824ca51 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 7f0ae9f)
@fkuner @svcnemo-autobot
Add Ling-V3 Tiny functional regression
e069d17 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 7ff158e)
@fkuner @yuzhongw-nvidia @svcnemo-autobot
Apply suggestions from code review
e239b61 
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
(cherry picked from commit 529e5e4)
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@fkuner @svcnemo-autobot
Rename Ling-V3 Tiny functional test
8c73427 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit c3bf414)
@fkuner @svcnemo-autobot
Fix MLA output gate compilation and tests
344b270 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit f08a989)
@fkuner @svcnemo-autobot
Validate GDN norm recompute for hybrid models
40f0423 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 545c761)
@fkuner @svcnemo-autobot
Tighten MLA output gate VJP tolerance
f6eba65 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 27f29bf)
@fkuner @svcnemo-autobot
Validate KDA safe-gate bounds
8ebbc56 
Reject missing or out-of-range lower bounds during KDA construction when safe gate is enabled, matching the FLA kernel contract.

Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 22458cd)
@yuzhongw-nvidia @svcnemo-autobot
refactor: Align KDA structure with GatedDeltaNet
f709ee9 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
(cherry picked from commit 4b03bfb)
@yuzhongw-nvidia @svcnemo-autobot
feat: Add attention output gate granularity
1a07060 
Signed-off-by: Yuzhong Wang <yuzhongw@nvidia.com>
(cherry picked from commit a7855d9)
@fkuner @svcnemo-autobot
Remove Ling-specific onboarding artifacts
de28a82 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit f9e20e5)
@fkuner @svcnemo-autobot
Remove FSDP DTensor support from Ling onboarding
ef8e9af 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit fe1caca)
@fkuner @svcnemo-autobot
Use the generic Hybrid spec in functional coverage
a7c70aa 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 611c8ac)
@fkuner @svcnemo-autobot
Refine KDA and hybrid FLOPs test contracts
0e06b45 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 1f6ffd1)
@fkuner @svcnemo-autobot
Correct MLA headwise gate VJP reference
c723eba 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 4a68b8a)
@fkuner @svcnemo-autobot
Correct KDA source attribution
7a31ac0 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 87c96f8)
@fkuner @yuzhongw-nvidia @svcnemo-autobot
Update megatron/training/training.py
8ff5307 
Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>
Signed-off-by: Fan Kun <784819644@qq.com>
(cherry picked from commit be824cc)
@fkuner @claude @svcnemo-autobot
Update megatron/core/transformer/multi_latent_attention.py
c7f07f7 
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Signed-off-by: Fan Kun <784819644@qq.com>
(cherry picked from commit aa516dc)
@fkuner @svcnemo-autobot
Fix hybrid layer count doctest examples
de4b538 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit bd98edc)
@fkuner @svcnemo-autobot
Fix Ling-V3 Tiny CI assertions
e51cb51 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit f4e113e)
@fkuner @svcnemo-autobot
Remove unused KDA parameter markers
2e6d0a6 
Signed-off-by: fkuner <784819644@qq.com>
(cherry picked from commit 14e28c6)
@svcnemo-autobot
chore: preserve current inference context while applying KDA
daddbfc 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge main into chore/cherrypick-14e28c6e-to-main-v2
18a826f 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix(ci): secure mamba recipe credentials
8b36b00 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix: adapt Ling-V3 imports to main
88a24d5 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
fix: restrict legacy checkpoint deserialization
bc17ee5 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
style: format gated delta net CP checks
f205386 
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
@svcnemo-autobot
Merge commit '21bbbe616db4858d46f348d6224bb56c5ddae1f7' into chore/ch…
91f6f85 
…errypick-14e28c6e-to-main-v2
@svcnemo-autobot
svcnemo-autobot requested review from a team as code owners August 29, 2026 12:42
@svcnemo-autobot
svcnemo-autobot requested a review from a team August 29, 2026 12:42
@svcnemo-autobot
svcnemo-autobot requested review from a team as code owners August 29, 2026 12:42
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft August 29, 2026 12:42
@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@kamran-nvidia kamran-nvidia mentioned this pull request Sep 29, 2026
Open
4 tasks done

This branch was previously deployed

2 inactive deployments
public — 91f6f85a Deployed Aug 29, 2026 by copy-pr-bot[bot] via build-test-publish-wheels / publish-wheels (megatron-core, arm64) #7719
test — 91f6f85a Deployed Aug 29, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #20439
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@svcnemo-autobot @svcnvidia-nemo-ci @fkuner @yuzhongw-nvidia