build(base): advance trainers-main to NVIDIA MCore 2a75ac12 and replay the 10 Baseten patches - #32
Closed
XiaohanZhangCMU wants to merge 290 commits into
Closed
build(base): advance trainers-main to NVIDIA MCore 2a75ac12 and replay the 10 Baseten patches#32XiaohanZhangCMU wants to merge 290 commits into
XiaohanZhangCMU wants to merge 290 commits into
Conversation
Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
…#5550) Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Markus Schnös <markus.schnoes@deepl.com> Co-authored-by: Markus Schnös <markus.schnoes@deepl.com>
…ky (NVIDIA#5848) Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
…IDIA#5798) Signed-off-by: Oleg Sudakov <osudakov@nvidia.com> Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com> Co-authored-by: Fei Wu <33940270+YangFei1990@users.noreply.github.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
…idden at checkpoint load time (NVIDIA#4705) Signed-off-by: John St John <jstjohn@nvidia.com> Signed-off-by: John St. John <jstjohn@nvidia.com>
…l" (NVIDIA#5324) Signed-off-by: Yury Parfenov <4665475+warpuv@users.noreply.github.com> Co-authored-by: Guihong Li <guihongl@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
…w pointer (NVIDIA#5639) Signed-off-by: Yan Bai <bayan@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Gautham Kollu <gkollu@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ckpointing (NVIDIA#5742) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…est (NVIDIA#5759) Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
…IDIA#5816) Signed-off-by: Gautham Kollu <gkollu@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
… failures (NVIDIA#5881) Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
…IA#5570) Signed-off-by: Yongqiang Wang <yongqiang.seagull@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com> Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com> Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
…VIDIA#6446) Signed-off-by: Jiangfei Duan <jiangfeid@nvidia.com>
…er size (NVIDIA#6379) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…IDIA#5590) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
NVIDIA#6469) Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Signed-off-by: Jiangfei Duan <jiangfeid@nvidia.com>
…6443) Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster> Signed-off-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Signed-off-by: Kezhi Kong <kezhik@nvidia.com> Co-authored-by: Kezhi Kong <devnkong@gmail.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Co-authored-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster> Co-authored-by: root <root@nvl72018-T18.cm.cluster> Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com> Co-authored-by: Kezhi Kong <kezhik@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster> Signed-off-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Signed-off-by: Kezhi Kong <kezhik@nvidia.com> Co-authored-by: Kezhi Kong <devnkong@gmail.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Co-authored-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster> Co-authored-by: root <root@nvl72018-T18.cm.cluster> Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com> Co-authored-by: Kezhi Kong <kezhik@nvidia.com>
Signed-off-by: Shiqing Fan <shiqingf@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com> Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
… linear dgrad
For a 3D grad_output, .matmul(weight) can be dispatched to a batched-GEMM
whose strideA argument is stored as int32 in the cuBLAS API. When
grad_output is a non-contiguous view (e.g. Megatron's standard [s, b, h]
layout on a [b, s, h]-contiguous storage), torch cannot collapse it to
2D without a copy and falls back to bmm. At long sequence and large
out-per-partition the resulting strideA = seq_len * out_per_partition
exceeds INT32_MAX and cuBLAS raises:
RuntimeError: at::cuda::blas::bgemm<at::BFloat16> argument ldb must
be positive and less than 2147483647 but got 2860646400
Repro: a frozen LM head under LoRA at seq=46080, vocab=248320, TP=4
(strideA = 46080 * 62080 = 2,860,646,400 > 2^31 - 1).
Flatten the leading dims into the M axis before the matmul so torch
routes through a single regular GEMM. The common Megatron-layout case
recovers the underlying [b, s, h]-contiguous view via a free
.transpose(0, 1) and the subsequent reshape becomes a pure view; for
any other 3D non-contiguous layout, fall back to an explicit reshape
that calls .contiguous() internally. The 2D path is unchanged.
Adds tests/unit_tests/tensor_parallel/test_layers.py
::test_LinearWithFrozenWeight_3d_non_contiguous_grad_output to defend
the dispatch path (the overflow itself only fires at sizes too large
for unit-test memory budgets; the test exercises the new code path at
small sizes against the same non-contiguous layout shape).
Signed-off-by: Kimbrian <kimbrian@parsed.com>
(cherry picked from commit 721e798)
(cherry picked from commit 6e90d30)
Restore the trainers-main GLM LoRA path on top of current NVIDIA main. Absorbed MLA reads linear_kv_up_proj as a raw weight because K is folded into the query and V is applied after core attention, so a normal AdapterWrapper forward would never run. Add a small override hook for the effective KV up-projection weight and use a GLM absorbed-MLA subclass that folds AdapterWrapper LoRA factors into that weight. The subclass is behaviorally identical when the projection is not LoRA-wrapped. LoRA on this absorbed KV up-projection remains limited to TP=1, matching the trainers-main support.
* fix(dsa): pass q_causal_offsets to the cuDNN indexer in packed-CP top-k paths
The cuDNN indexer_forward_wrapper applies TOP-LEFT-aligned causal masking by
default (row i keeps keys j <= i), but both packed-CP indexer top-k paths hand
it query chunks whose rows sit at ABSOLUTE causal positions inside a key
prefix cropped to key_end:
- _indexer_topk_from_score_chunks (single packed THD sequence, CP front/back
segments): each row chunk's q[0] sits at global key index
bottom_right_key_start + row_start;
- _indexer_topk_multi_packed_cp_thd (multi-document packed THD): each THD
segment's q[0] sits at doc-local key index
segment_k_lengths - segment_q_lengths.
Without the offsets, every zigzag chunk except cp_rank 0's front chunk is
masked to a chunk-local window (~seq/2cp keys) instead of its true causal
prefix, so the downstream top-k silently selects from the wrong keys
(measured 2-25% overlap vs an exact fp32 torch reference at cp_size=32,
GLM-5.2 indexer dims), and at 131k tokens the mismatched -inf pattern
surfaces as cudaErrorIllegalAddress inside indexer_top_k. Training does NOT
crash at short sequence lengths - it just learns on a wrong sparse-attention
pattern.
Passing the kernel's q_causal_offsets argument ("global uncompressed token
index for each batch/THD segment's local q[0]", cudnn 1.25.0) at both call
sites makes index parity exact (overlap 1.0000, zero causally-out-of-bounds
indices) vs the torch reference for single-doc rows at 8k/32k/131k across
cp ranks 0/15/31 and multi-doc packs [8192,4096]/[65536,65536]/[131008,64],
and eliminates the 131k IMA (reproduced standalone at docs=[65536,65536],
cp_rank=31 before the fix).
Note test_cudnn_indexer_topk_single_packed_cp_real_kernel_uses_bottom_right_alignment
(the one test that runs the real kernel on this path) is currently disabled as
flaky (cutlass ThrMma build issue); the remaining tests mock the kernel and
mask this defect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(dsa): clarify packed CP causal offset comments
Keep the per-path invariants close to the code without duplicating the
failure-mode explanation at both call sites.
Signed-off-by: Jack Rao <jack.rao@baseten.co>
* docs(dsa): remove redundant causal offset comments
Keep the packed-CP rationale in the LM#14 description instead of duplicating
it beside both call sites.
Signed-off-by: Jack Rao <jack.rao@baseten.co>
---------
Signed-off-by: Jack Rao <jack.rao@baseten.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Keep fused score generation and sparse attention while selecting odd top-k values with PyTorch, avoiding cuDNN Frontend's vector-width assertion for packed CP segments. Co-authored-by: Cursor <cursoragent@cursor.com>
[tool.uv.sources] entries propagate to consumers that vendor this repo as a path dependency; the git pin overrode their prebuilt-wheel routing for fht. The plain requirement stays; consumers pick the source (the dependency-metadata stanza keeps lock-anywhere working).
#25) Released TE's pad_between_seqs auto-detect ignores padding after the last sequence; under context parallelism that silently corrupts chunk-boundary rows (nondeterministic forward/gradients + wrong attention). Compute the tail-inclusive answer at the call site and pass it explicitly. Details: #25. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The ~=0.4.0 cap excluded all of 0.5.x, making megatron-bridge's flash-linear-attention>=0.5.2 floor (needed by Kimi-K3's KDA kernel) unsatisfiable. MCore's own fla surface -- causal_conv1d, l2norm, chunk_gated_delta_rule in ssm/gated_delta_net.py -- is signature-compatible across 0.4.2 and 0.5.2, so this only stops MCore from excluding 0.5.x; it does not require it.
`get_transformer_layer_offset(config, vp_stage, pp_rank)` accepts a rank and every branch uses it except the custom-layout one, which called `PipelineParallelLayerLayout.get_layer_offset` without passing it on. That helper then fell back to the local rank, so asking for another stage's first layer returned this stage's answer. `get_layer_offset` already takes `pp_rank`, so this forwards the argument the caller supplied. It matters for callers that enumerate every stage's first layer from one process. Kimi-K3 does exactly that to decide which layers sit on a pipeline boundary and therefore have to pack the AttnRes snapshot bank into the payload: with the offsets collapsed to one value, the sending and receiving stages disagree about the payload width. Signed-off-by: Xiaohan Zhang <xiaohan.zhang@baseten.co>
Author
|
Closing. The approach here was wrong: I targeted The right shape is to cherry-pick our |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Advance
trainers-mainto NVIDIA MCore2a75ac12(2026-08-13) and replay our 10 Baseten patches on top, so the history reads[nvidia main 0..t][our patches]instead of our patches sitting on a base from 2026-07-05.This is the MCore half of the Megatron-Bridge base bump. The bridge cannot move without it: upstream's current bridge expects an MCore our fork was 280 commits behind, while carrying 10 patches upstream does not have.
Why now
The Kimi-K3 work needed a bridge newer than our fork's base, and cherry-picking it onto an old base left the K3 modelling code running against a tree three weeks older than the one it was written for. Bumping the base instead makes that mismatch go away, and keeps the lineage clean for the next model that lands upstream first.
What replayed
All 10 patches, unchanged in intent:
fix(tensor-parallel): avoid int32 stride overflow in frozen linear dgradmoe: clear dispatcher forward state after combinefix(mla): use int64 row offsets in fused RoPE kernelsfix(dsa): align packed-CP indexer causal masks (#16)fix(dsa): fall back for unsupported odd cuDNN top-k (#19)build: drop the fast-hadamard-transform git source pinfix(te): passpad_between_seqsexplicitly for tail-padded THD under CP (#25)build(deps): relax the flash-linear-attention ceilingfix(pp): honour the explicitpp_rankunder a custom pipeline layoutThe three conflicts, and how each was resolved
tensor_parallel/layers.py- upstream had addedgrad_output = grad_output.to(ctx.input_dtype)ahead of thedim() > 2dispatch our patch replaces. Kept upstream's cast and ourdim() == 3fast path, so both intents survive. Verified by count: the cast still appears at both of its upstream call sites.tests/unit_tests/tensor_parallel/test_layers.py- purely additive on both sides, no overlapping test names. Union: upstream's three FP32-output tests plus ourtest_LinearWithFrozenWeight_3d_non_contiguous_grad_output.transformer/moe/token_dispatcher.py- two additive blocks. Union: upstream'sget_expert_zero_copy_buffersplus our_clear_forward_state, and our five= Noneresets in_HybridEPManager.pyproject.toml- the one resolution that is not a union. Upstream now pinsflash-linear-attention==0.5.1, which is below the floor Kimi-K3 needs: before 0.5.2,chunk_kdaacceptsA_loganddt_biasthrough**kwargsand silently discards them, training a different KDA forget gate with nothing raised. Resolved to>=0.5.2,<0.6- upstream's 0.5.x line, floored where K3 becomes correct. This conflict did not exist on the old base, and it is the clearest argument for bumping rather than cherry-picking.A warning for anyone repeating this
Do not resolve this replay with
-X theirs. It exits zero, reports every patch applied, and silently deletes upstream lines our patches never touched - it dropped one of the twoto(ctx.input_dtype)casts here, and Ian'sasync_ckpt_use_cpu_shmon the bridge side. Every conflict above was resolved by hand and then checked per file for both directions: our intent present, no upstream line missing.Testing
Not run: MCore's suite needs GPUs. Static checks only - every changed file parses, no conflict markers, and each patch's marker symbols are present (
int32 strideAcomment,_clear_forward_state,pad_between_seqs,get_expert_zero_copy_buffers).The real gate is the validation matrix on the paired bridge PR, which has to cover GLM-5.2 THD-CP, DSv4-Flash, MiMo and Qwen as well as Kimi-K3, because these patches serve all of them.
Landing this: GitHub will say CONFLICTING, and that is correct
A base bump does not merge into the branch it replaces - it is the new branch. Merging would ask git to reconcile our replayed patches with the pre-replay originals still on
trainers-main, and the only conflicts are the submodule gitlink and the files our own patches touch on both sides.Do not resolve it with a merge commit. I tried that first (
-s ours, which keeps the right tree) and it dragged the entire oldtrainers-mainline in as a second parent: 31 commits above upstream instead of 13, with every Baseten patch appearing twice - once replayed, once as the pre-replay original. That is exactly the muddied lineage this PR exists to remove, so it is reverted.Land it by pointing
trainers-mainat this branch:The previous tip is preserved as
backup/trainers-main-20260819, matching the existingbackup/trainers-main-20260704convention.Verifying the history is clean
Every commit above the upstream tip is authored by a Baseten engineer, none rewrite upstream history, and there are no merge commits.