Skip to content

build(base): advance trainers-main to NVIDIA MCore 2a75ac12 and replay the 10 Baseten patches - #32

Closed
XiaohanZhangCMU wants to merge 290 commits into
trainers-mainfrom
mcore/replay-onto-upstream-20260813
Closed

build(base): advance trainers-main to NVIDIA MCore 2a75ac12 and replay the 10 Baseten patches#32
XiaohanZhangCMU wants to merge 290 commits into
trainers-mainfrom
mcore/replay-onto-upstream-20260813

Conversation

@XiaohanZhangCMU

@XiaohanZhangCMU XiaohanZhangCMU commented Aug 19, 2026

Copy link
Copy Markdown

Advance trainers-main to NVIDIA MCore 2a75ac12 (2026-08-13) and replay our 10 Baseten patches on top, so the history reads [nvidia main 0..t][our patches] instead of our patches sitting on a base from 2026-07-05.

This is the MCore half of the Megatron-Bridge base bump. The bridge cannot move without it: upstream's current bridge expects an MCore our fork was 280 commits behind, while carrying 10 patches upstream does not have.

Why now

The Kimi-K3 work needed a bridge newer than our fork's base, and cherry-picking it onto an old base left the K3 modelling code running against a tree three weeks older than the one it was written for. Bumping the base instead makes that mismatch go away, and keeps the lineage clean for the next model that lands upstream first.

What replayed

All 10 patches, unchanged in intent:

Patch Author
fix(tensor-parallel): avoid int32 stride overflow in frozen linear dgrad Kimbrian
moe: clear dispatcher forward state after combine Kimbrian
fix(mla): use int64 row offsets in fused RoPE kernels Paras
support LoRA on absorbed GLM MLA Paras
fix(dsa): align packed-CP indexer causal masks (#16) Jack
fix(dsa): fall back for unsupported odd cuDNN top-k (#19) Jack
build: drop the fast-hadamard-transform git source pin Kimbrian
fix(te): pass pad_between_seqs explicitly for tail-padded THD under CP (#25) Jack
build(deps): relax the flash-linear-attention ceiling Xiaohan
fix(pp): honour the explicit pp_rank under a custom pipeline layout Xiaohan

The three conflicts, and how each was resolved

tensor_parallel/layers.py - upstream had added grad_output = grad_output.to(ctx.input_dtype) ahead of the dim() > 2 dispatch our patch replaces. Kept upstream's cast and our dim() == 3 fast path, so both intents survive. Verified by count: the cast still appears at both of its upstream call sites.

tests/unit_tests/tensor_parallel/test_layers.py - purely additive on both sides, no overlapping test names. Union: upstream's three FP32-output tests plus our test_LinearWithFrozenWeight_3d_non_contiguous_grad_output.

transformer/moe/token_dispatcher.py - two additive blocks. Union: upstream's get_expert_zero_copy_buffers plus our _clear_forward_state, and our five = None resets in _HybridEPManager.

pyproject.toml - the one resolution that is not a union. Upstream now pins flash-linear-attention==0.5.1, which is below the floor Kimi-K3 needs: before 0.5.2, chunk_kda accepts A_log and dt_bias through **kwargs and silently discards them, training a different KDA forget gate with nothing raised. Resolved to >=0.5.2,<0.6 - upstream's 0.5.x line, floored where K3 becomes correct. This conflict did not exist on the old base, and it is the clearest argument for bumping rather than cherry-picking.

A warning for anyone repeating this

Do not resolve this replay with -X theirs. It exits zero, reports every patch applied, and silently deletes upstream lines our patches never touched - it dropped one of the two to(ctx.input_dtype) casts here, and Ian's async_ckpt_use_cpu_shm on the bridge side. Every conflict above was resolved by hand and then checked per file for both directions: our intent present, no upstream line missing.

Testing

Not run: MCore's suite needs GPUs. Static checks only - every changed file parses, no conflict markers, and each patch's marker symbols are present (int32 strideA comment, _clear_forward_state, pad_between_seqs, get_expert_zero_copy_buffers).

The real gate is the validation matrix on the paired bridge PR, which has to cover GLM-5.2 THD-CP, DSv4-Flash, MiMo and Qwen as well as Kimi-K3, because these patches serve all of them.

Landing this: GitHub will say CONFLICTING, and that is correct

A base bump does not merge into the branch it replaces - it is the new branch. Merging would ask git to reconcile our replayed patches with the pre-replay originals still on trainers-main, and the only conflicts are the submodule gitlink and the files our own patches touch on both sides.

Do not resolve it with a merge commit. I tried that first (-s ours, which keeps the right tree) and it dragged the entire old trainers-main line in as a second parent: 31 commits above upstream instead of 13, with every Baseten patch appearing twice - once replayed, once as the pre-replay original. That is exactly the muddied lineage this PR exists to remove, so it is reverted.

Land it by pointing trainers-main at this branch:

git fetch origin
git push --force-with-lease origin origin/<this-branch>:refs/heads/trainers-main

The previous tip is preserved as backup/trainers-main-20260819, matching the existing backup/trainers-main-20260704 convention.

Verifying the history is clean

git merge-base --is-ancestor <upstream-tip> <this-branch>   # upstream untouched
git log --oneline <upstream-tip>..<this-branch>             # only our commits
git log --merges --oneline <upstream-tip>..<this-branch>    # empty

Every commit above the upstream tip is authored by a Baseten engineer, none rewrite upstream history, and there are no merge commits.

ko3n1g and others added 30 commits July 16, 2026 10:23
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: oliver könig <okoenig@nvidia.com>
)

Signed-off-by: Siddharth Singh <sidsingh@nvidia.com>
…#5550)

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Markus Schnös <markus.schnoes@deepl.com>
Co-authored-by: Markus Schnös <markus.schnoes@deepl.com>
…ky (NVIDIA#5848)

Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
…IDIA#5798)

Signed-off-by: Oleg Sudakov <osudakov@nvidia.com>
Signed-off-by: Oleg Sudakov <oleg.sudakov@outlook.com>
Co-authored-by: Fei Wu <33940270+YangFei1990@users.noreply.github.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
…idden at checkpoint load time (NVIDIA#4705)

Signed-off-by: John St John <jstjohn@nvidia.com>
Signed-off-by: John St. John <jstjohn@nvidia.com>
…l" (NVIDIA#5324)

Signed-off-by: Yury Parfenov <4665475+warpuv@users.noreply.github.com>
Co-authored-by: Guihong Li <guihongl@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
…w pointer (NVIDIA#5639)

Signed-off-by: Yan Bai <bayan@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Gautham Kollu <gkollu@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ckpointing (NVIDIA#5742)

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…est (NVIDIA#5759)

Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Lawrence McAfee <lmcafee@nvidia.com>
… failures (NVIDIA#5881)

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
)

Signed-off-by: Evgenii Zheltonozhskii <zheltonozhskiy@gmail.com>
Co-authored-by: Maanu Grover <maanug@nvidia.com>
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
…IA#5570)

Signed-off-by: Yongqiang Wang <yongqiang.seagull@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Achyuthan Sivasankar <achyuthan.sivasankar@gmail.com>
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
github-actions Bot and others added 25 commits August 12, 2026 09:26
…er size (NVIDIA#6379)

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…IDIA#5590)

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>
Signed-off-by: Jiangfei Duan <jiangfeid@nvidia.com>
…6443)

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
Signed-off-by: Kezhi Kong <kezhik@nvidia.com>
Co-authored-by: Kezhi Kong <devnkong@gmail.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
Co-authored-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: root <root@nvl72018-T18.cm.cluster>
Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com>
Co-authored-by: Kezhi Kong <kezhik@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Philip Petrakian <ppetrakian@nvidia.com>
Signed-off-by: Maanu Grover <maanug@nvidia.com>
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
Signed-off-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster>
Signed-off-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
Signed-off-by: Kezhi Kong <kezhik@nvidia.com>
Co-authored-by: Kezhi Kong <devnkong@gmail.com>
Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com>
Co-authored-by: Mikail Khona <mkhona@cw-dfw-cs-001-vscode-01.cm.cluster>
Co-authored-by: root <root@nvl72018-T18.cm.cluster>
Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com>
Co-authored-by: Kezhi Kong <kezhik@nvidia.com>
Signed-off-by: Shiqing Fan <shiqingf@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Co-authored-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
Signed-off-by: Deyu Fu <deyuf@nvidia.com>
… linear dgrad

For a 3D grad_output, .matmul(weight) can be dispatched to a batched-GEMM
whose strideA argument is stored as int32 in the cuBLAS API. When
grad_output is a non-contiguous view (e.g. Megatron's standard [s, b, h]
layout on a [b, s, h]-contiguous storage), torch cannot collapse it to
2D without a copy and falls back to bmm. At long sequence and large
out-per-partition the resulting strideA = seq_len * out_per_partition
exceeds INT32_MAX and cuBLAS raises:

    RuntimeError: at::cuda::blas::bgemm<at::BFloat16> argument ldb must
    be positive and less than 2147483647 but got 2860646400

Repro: a frozen LM head under LoRA at seq=46080, vocab=248320, TP=4
(strideA = 46080 * 62080 = 2,860,646,400 > 2^31 - 1).

Flatten the leading dims into the M axis before the matmul so torch
routes through a single regular GEMM. The common Megatron-layout case
recovers the underlying [b, s, h]-contiguous view via a free
.transpose(0, 1) and the subsequent reshape becomes a pure view; for
any other 3D non-contiguous layout, fall back to an explicit reshape
that calls .contiguous() internally. The 2D path is unchanged.

Adds tests/unit_tests/tensor_parallel/test_layers.py
::test_LinearWithFrozenWeight_3d_non_contiguous_grad_output to defend
the dispatch path (the overflow itself only fires at sizes too large
for unit-test memory budgets; the test exercises the new code path at
small sizes against the same non-contiguous layout shape).

Signed-off-by: Kimbrian <kimbrian@parsed.com>
(cherry picked from commit 721e798)
Restore the trainers-main GLM LoRA path on top of current NVIDIA main.

Absorbed MLA reads linear_kv_up_proj as a raw weight because K is folded into the query and V is applied after core attention, so a normal AdapterWrapper forward would never run. Add a small override hook for the effective KV up-projection weight and use a GLM absorbed-MLA subclass that folds AdapterWrapper LoRA factors into that weight.

The subclass is behaviorally identical when the projection is not LoRA-wrapped. LoRA on this absorbed KV up-projection remains limited to TP=1, matching the trainers-main support.
* fix(dsa): pass q_causal_offsets to the cuDNN indexer in packed-CP top-k paths

The cuDNN indexer_forward_wrapper applies TOP-LEFT-aligned causal masking by
default (row i keeps keys j <= i), but both packed-CP indexer top-k paths hand
it query chunks whose rows sit at ABSOLUTE causal positions inside a key
prefix cropped to key_end:

- _indexer_topk_from_score_chunks (single packed THD sequence, CP front/back
  segments): each row chunk's q[0] sits at global key index
  bottom_right_key_start + row_start;
- _indexer_topk_multi_packed_cp_thd (multi-document packed THD): each THD
  segment's q[0] sits at doc-local key index
  segment_k_lengths - segment_q_lengths.

Without the offsets, every zigzag chunk except cp_rank 0's front chunk is
masked to a chunk-local window (~seq/2cp keys) instead of its true causal
prefix, so the downstream top-k silently selects from the wrong keys
(measured 2-25% overlap vs an exact fp32 torch reference at cp_size=32,
GLM-5.2 indexer dims), and at 131k tokens the mismatched -inf pattern
surfaces as cudaErrorIllegalAddress inside indexer_top_k. Training does NOT
crash at short sequence lengths - it just learns on a wrong sparse-attention
pattern.

Passing the kernel's q_causal_offsets argument ("global uncompressed token
index for each batch/THD segment's local q[0]", cudnn 1.25.0) at both call
sites makes index parity exact (overlap 1.0000, zero causally-out-of-bounds
indices) vs the torch reference for single-doc rows at 8k/32k/131k across
cp ranks 0/15/31 and multi-doc packs [8192,4096]/[65536,65536]/[131008,64],
and eliminates the 131k IMA (reproduced standalone at docs=[65536,65536],
cp_rank=31 before the fix).

Note test_cudnn_indexer_topk_single_packed_cp_real_kernel_uses_bottom_right_alignment
(the one test that runs the real kernel on this path) is currently disabled as
flaky (cutlass ThrMma build issue); the remaining tests mock the kernel and
mask this defect.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(dsa): clarify packed CP causal offset comments

Keep the per-path invariants close to the code without duplicating the
failure-mode explanation at both call sites.

Signed-off-by: Jack Rao <jack.rao@baseten.co>

* docs(dsa): remove redundant causal offset comments

Keep the packed-CP rationale in the LM#14 description instead of duplicating
it beside both call sites.

Signed-off-by: Jack Rao <jack.rao@baseten.co>

---------

Signed-off-by: Jack Rao <jack.rao@baseten.co>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Keep fused score generation and sparse attention while selecting odd top-k values with PyTorch, avoiding cuDNN Frontend's vector-width assertion for packed CP segments.

Co-authored-by: Cursor <cursoragent@cursor.com>
[tool.uv.sources] entries propagate to consumers that vendor this repo
as a path dependency; the git pin overrode their prebuilt-wheel routing
for fht. The plain requirement stays; consumers pick the source (the
dependency-metadata stanza keeps lock-anywhere working).
#25)

Released TE's pad_between_seqs auto-detect ignores padding after the last
sequence; under context parallelism that silently corrupts chunk-boundary
rows (nondeterministic forward/gradients + wrong attention). Compute the
tail-inclusive answer at the call site and pass it explicitly.

Details: #25.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The ~=0.4.0 cap excluded all of 0.5.x, making megatron-bridge's
flash-linear-attention>=0.5.2 floor (needed by Kimi-K3's KDA kernel)
unsatisfiable. MCore's own fla surface -- causal_conv1d, l2norm,
chunk_gated_delta_rule in ssm/gated_delta_net.py -- is signature-compatible
across 0.4.2 and 0.5.2, so this only stops MCore from excluding 0.5.x; it
does not require it.
`get_transformer_layer_offset(config, vp_stage, pp_rank)` accepts a rank and
every branch uses it except the custom-layout one, which called
`PipelineParallelLayerLayout.get_layer_offset` without passing it on. That
helper then fell back to the local rank, so asking for another stage's first
layer returned this stage's answer.

`get_layer_offset` already takes `pp_rank`, so this forwards the argument the
caller supplied.

It matters for callers that enumerate every stage's first layer from one
process. Kimi-K3 does exactly that to decide which layers sit on a pipeline
boundary and therefore have to pack the AttnRes snapshot bank into the payload:
with the offsets collapsed to one value, the sending and receiving stages
disagree about the payload width.

Signed-off-by: Xiaohan Zhang <xiaohan.zhang@baseten.co>
@XiaohanZhangCMU

Copy link
Copy Markdown
Author

Closing. The approach here was wrong: I targeted 2a75ac12 (the specific commit the new bridge pins) and hit three conflicts, which is a sign the base was picked for convenience rather than for lineage.

The right shape is to cherry-pick our trainers-main patches onto Megatron-LM's main, giving a linear NVIDIA-then-Baseten history with no conflicts. Reopening as a fresh PR once that is prepared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.