[build] chore: pin CPU-only conversion fixes for validation

[cuichenx]

Summary

What

• Stack the CPU-import Gloo fix from fix(conversion): initialize CPU imports with Gloo #5313.
• Pin the Megatron-Core submodule to the exact public head of
  fix: avoid MCore CUDA assumptions in CPU-only model conversion NVIDIA/Megatron-LM#6270 so full GPT-OSS CPU-import validation does not use a
  local override.
• Stack the CPU-only GPT layer-spec fix from [model] fix: use local GPT spec for CPU-only initialization #5317 via a signed merge commit.
• Stack the MCore FSDP factory compatibility fix from [model] fix: support MCore FSDP factory wrappers #5322 via its public
  signed/DCO commit.
• Stack the disabled-RNG checkpoint fix from [ckpt] fix: honor disabled RNG saves #5323 via its public signed/DCO
  commit.
• Pin the MCore CPU-only initialization and checkpoint fixes from
  fix: avoid MCore CUDA assumptions in CPU-only model conversion NVIDIA/Megatron-LM#6270, including the local/TE MoE checkpoint-key mapping
  and empty TE norm/attention extra-state compatibility corrections exposed by
  the full H100 reloads.
• Preserve strict GPT-OSS HF export while mapping each dequantized expert
  gate_up_proj/down_proj output key to the physical MXFP4 block and scale
  source keys that it replaces.

This is a temporary stacked integration PR. The current exact-public head has
passed the full zero-GPU CPU import plus H100/Transformer-Engine strict
load/export gate. Forward and numerical parity are not claimed by that gate.

Dependency delta

Megatron-LM 6513e3e23d6b5eda6a1c934990b15e804237732b
         -> d57d882fd08ee1762eab6144260cf9693f58ac9d (NVIDIA/Megatron-LM#6270)

Validation

• Bridge focused unit tests: 7 passed on fix(conversion): initialize CPU imports with Gloo #5313.
• MCore YARN, attention, MoE-dispatcher, checkpoint-preload, and finalizer
  focused tests from exact public MCore PR head
  c081d1a5c1b7aad72614e701ad39b67c8762e50b on all 8 ranks in
  nvcr.io/nvidian/nemo:26.08.rc4: every rank reported 9 passed (DFW job
  15105873, COMPLETED 0:0).
• Dense/MoE checkpoint-key regression cases from a fresh clone of exact
  public MCore PR head 4220c840f520f621ca73415c8e73212998856b3c:
  both cases passed on all 8 ranks in nvcr.io/nvidian/nemo:26.08.rc4 (DFW job
  15108027, COMPLETED 0:0).
• Dense/MoE key and empty/non-empty TE norm extra-state compatibility cases
  from a fresh clone of exact public MCore PR head
  9ba6247b7495c5a2cc1692ca8d14fd21947f305e: all four passed on all 8 H100
  ranks in nvcr.io/nvidian/nemo:26.08.rc4 (DFW job 15110325, COMPLETED 0:0).
• Dense/MoE key and TE norm/attention extra-state compatibility cases from
  a clean detached worktree of exact public MCore PR head
  d57d882fd08ee1762eab6144260cf9693f58ac9d: all five passed on all 8 H100
  ranks in nvcr.io/nvidian/nemo:26.08.rc4 (DFW job 15112718, COMPLETED 0:0); the fetched PR ref, exact SHA, and clean tree were rechecked afterward.
• Conversion utility, GPT provider, and disabled-RNG checkpoint focused
  tests: 38 passed against exact pinned MCore d57d882f... in driverless
  nvcr.io/nvidia/nemo:26.06.01 at pushed Bridge head
  99433894a7c6ec98a25079e123038b60cdff53c3.
• GPT-OSS replacement-map and ordinary/distributed strict-saver focused
  tests: 34 passed against the mounted candidate checkout; all-file pre-commit
  passed. Independent exact-diff review found no blocking findings.
• uv lock --check.
• uv run --no-sync pre-commit run --all-files.
• Full openai/gpt-oss-20b CPU import from exact clean pushed head
  027e284beff1e6b5d7ddb1b2ebebc462a9c2bc5b: DFW job 15108616 completed
  0:0 on genuine zero-GPU node cpu1-00055, loaded 20,914,757,184
  parameters, and produced the complete 41,839,126,873-byte torch-dist shard.
• The one-H100/TE retry 15109663 reached the complete checkpoint plan and
  cleared the earlier MoE weight-key defect, then exposed missing optional
  empty TE norm extra state. No output was produced; the clean exact-public
  failure is fixed by MCore head 9ba6247b... above.
• Full openai/gpt-oss-20b CPU import from exact clean pushed Bridge head
  dbe0a5b7a4c692f8dc8695fcb53601c7edbfc02d / MCore 9ba6247b...: DFW job
  15110835 completed 0:0 on genuine zero-GPU node cpu1-00035, loaded all
  20,914,757,184 parameters, and produced a complete 41,839,125,360-byte shard.
• One-H100/TE job 15111919 consumed only that fresh checkpoint, cleared
  the prior MoE key and TE norm-state defects, then exposed missing optional
  self_attention.core_attention._extra_state. No output was created; the
  clean exact-public failure is fixed by MCore d57d882f... above.
• Full openai/gpt-oss-20b CPU import from exact clean public Bridge head
  d3346db7c4a8eedd43311d85d888b39687469f22 / MCore d57d882f...: DFW job
  15114882 completed 0:0 in 18m19s on genuine zero-GPU node cpu1-00035,
  loaded all 20,914,757,184 parameters, and produced a fresh
  41,839,125,360-byte torch-dist shard.
• One-H100/TE strict load/export job 15115717 consumed only that fresh
  checkpoint in nvcr.io/nvidian/nemo:26.08.rc4 and completed 0:0 in 5m44s
  on pool0-01779. It constructed the full TE model, converted all 3,291
  Bridge tasks, and wrote a strict HF index with 411 dequantized tensor keys
  across three shards, no expert MXFP4 block/scale keys, and no missing
  referenced shard. Both exact-public trees remained clean afterward.

No tests will be quarantined as part of this investigation without Chen's
explicit approval. If the full import exposes another bug, that bug will be
reported and fixed in its owning upstream repository before a clean pushed
rerun; it will not be hidden with a local patch.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[cuichenx]

Full openai/gpt-oss-20b CPU import from exact clean pushed head 781a6162b14edea3d74e5aef83b659523e9964ba failed at the next independent blocker.

The run cleared the Gloo initialization fix and the YARN CPU-initialization fix from NVIDIA/Megatron-LM#6270, then failed during GPT layer construction because the default provider selected Transformer Engine DotProductAttention on a no-CUDA host (RuntimeError: TransformerEngine needs CUDA). No checkpoint or partial output was created, and both source checkouts remained clean at the exact public commits after failure.

This workflow remains not verified. A previous local-spec approach in #3853 was closed over compatibility concerns between conversion-time local modules and later GPU/TE models, so I am not silently reusing it. The next fix needs explicit parameter-schema and GPU-load compatibility validation.

[cuichenx]

/ok to test 4b4bb03

[cuichenx]

The full GPT-OSS CPU import from exact clean public head 4b4bb03c8c614fe794d9ecf3f12270072809a025 selected the MCore-local GPT spec successfully, then failed at the next independent blocker: DotProductAttention allocated its softmax offset through torch.cuda.current_device() despite use_cpu_initialization=True.

No checkpoint or partial output was produced, and the Bridge and MCore checkouts remained clean at the exact public commits after failure. The owning MCore fix is now public on NVIDIA/Megatron-LM#6270 at 14450c8af9c8d4e5214b08f6bf00f2fa1b780521 with focused tests.

This PR remains not verified. Its MCore gitlink will be advanced to that public commit, then the full import will be rerun from a fresh exact public clone; no local source patch or overlay will be used as evidence.

[cuichenx]

MCore PR #6270 is now mirrored publicly at exact head 14450c8af9c8d4e5214b08f6bf00f2fa1b780521. From a fresh clean clone, all eight ranks passed all four focused YARN and attention CPU-initialization tests in nvcr.io/nvidian/nemo:26.08.rc4.

This integration PR now pins that exact commit at Bridge head 49700fa8a00d9a306a2dfd2637ee5e695511423f; uv.lock was regenerated for the new MCore metadata. Bridge provider tests pass 29/29 in a driverless 26.06 container, uv lock --check passes, and all-files pre-commit passes.

The full GPT-OSS import is still not verified. The next evidence will be a fresh clean clone of this exact pushed head running the unchanged full workload.

[cuichenx]

/ok to test 49700fa

[cuichenx]

The unchanged full GPT-OSS CPU import from exact clean public Bridge head 49700fa8a00d9a306a2dfd2637ee5e695511423f and MCore head 14450c8af9c8d4e5214b08f6bf00f2fa1b780521 progressed past the Gloo, YARN, local-spec, and attention-offset blockers. It then failed at the next independent MCore defect: MoEAlltoAllTokenDispatcher allocated fused-permutation metadata on CUDA during CPU-only construction.

No checkpoint or partial output was produced. Both source trees remained clean and unchanged at the exact public commits after failure. The owning fix is now pushed on NVIDIA/Megatron-LM#6270 at 593bc0aa8ab28cd4d4e2bb44426280f5f1e4d787 with a focused driverless regression test.

This PR remains not verified. After public mirror and focused validation, its MCore gitlink will be advanced and the unchanged full import rerun from a new clean public clone.

[cuichenx]

Advanced the integration branch normally (no history rewrite) to Bridge head
5c8bc9c257c6af5648b14a0d3d032e7067ae1868, which pins exact public mirrored
MCore PR #6270 head 593bc0aa8ab28cd4d4e2bb44426280f5f1e4d787.

Focused checks on this exact Bridge state:

• driverless GPT provider tests: 29 passed in nvcr.io/nvidia/nemo:26.06.01
• uv lock --check: passed
• uv run --no-sync pre-commit run --all-files: passed

An exact-public 8-rank MCore focused run is registered as DFW job 15099712.
The unchanged full GPT-OSS CPU import will be rerun from a fresh clone only
after this pushed integration state is established. No full-workflow
verification pass is claimed.

[cuichenx]

/ok to test 5c8bc9c

[cuichenx]

The exact-public 8-rank MCore focused run passed, and the unchanged full
GPT-OSS import is now running from a new clean clone of this PR head.

• Bridge: 5c8bc9c257c6af5648b14a0d3d032e7067ae1868
• MCore: 593bc0aa8ab28cd4d4e2bb44426280f5f1e4d787
• DFW job: 15099928
• node/resources: genuine zero-GPU cpu1-00140, one task, 192 GiB, null GPU
  resources, shared placement
• image/workload: nvcr.io/nvidia/nemo:26.06.01, unchanged full
  openai/gpt-oss-20b CPU BF16 TP1/PP1/EP1/ETP1 import

Both source trees were clean and reachable from their public refs immediately
before submission. This is candidate validation; no verification pass is
claimed while the job is running.

[cuichenx]

Full exact-public job 15099928 failed 1:0 after 7m36s at the next Bridge
blocker. No checkpoint or partial output was created.

The public stack cleared the prior Gloo, YARN, local-spec, attention-offset,
and MoE-dispatcher failures and reached HF weight loading. Bridge then failed in
models/conversion/utils.py::unwrap_model():

TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union

Current MCore changed mcore_fsdp_adapter.FullyShardedDataParallel from a class
to a V1/V2 factory in NVIDIA/Megatron-LM#5865, while Bridge's custom
unwrap_model() still includes that factory function in its default
isinstance() tuple. Current MCore's own unwrap_model() was updated to use
the concrete V1/V2 wrapper classes, but Bridge's extension was not.

Post-run provenance checks passed: Bridge remained clean at
5c8bc9c257c6af5648b14a0d3d032e7067ae1868, MCore remained clean at
593bc0aa8ab28cd4d4e2bb44426280f5f1e4d787, and the output root is empty.
This is a failed candidate validation, not a verification pass. The fix will be
published in an owning Bridge PR before the unchanged workload is rerun from a
new exact-public clone.

[cuichenx]

Advanced this integration PR normally to signed/DCO head dbe0a5b7a4c692f8dc8695fcb53601c7edbfc02d, pinning exact public MCore PR #6270 head 9ba6247b7495c5a2cc1692ca8d14fd21947f305e. No public history was rewritten.

• exact-public MCore job 15110325: COMPLETED 0:0; all eight H100 ranks reported all four dense/MoE key and TE norm extra-state compatibility cases passed
• Bridge conversion, GPT-provider, and disabled-RNG checkpoint tests: 38 passed in driverless nvcr.io/nvidia/nemo:26.06.01
• uv lock --check: passed
• uv run --no-sync pre-commit run --all-files: passed
• GitHub commit signature: valid; DCO: green

For completeness, a broader driverless full-file diagnostic collected 263 tests and reported 262 passed plus one existing NCCL-only mock test failure (TestTemporaryDistributedContext::test_temporary_distributed_context_nccl) because the no-driver container makes torch.cuda.is_available() false. That result is recorded rather than hidden, but it is not one of the 38 scoped integration gates and is unrelated to this gitlink-only change. It needs a separate upstream test-harness correction; no test is being quarantined.

The previous full CPU checkpoint from head 027e284b... will not be reused as final proof. I will generate a new full GPT-OSS 20B checkpoint from a fresh clean clone of this exact pushed head, then load/export it with the real H100/TE model. Until both complete, the workflow remains not verified.

[cuichenx]

/ok to test dbe0a5b

[cuichenx]

Fresh exact-public H100/TE job 15111919 failed on the next upstream checkpoint-compatibility defect after clearing both prior MoE norm failures.

• exact Bridge head: dbe0a5b7a4c692f8dc8695fcb53601c7edbfc02d
• exact MCore head: 9ba6247b7495c5a2cc1692ca8d14fd21947f305e
• input: the complete new 41,839,125,360-byte distcp checkpoint produced by zero-GPU job 15110835 from those same heads
• workload: full 20,914,757,184-parameter GPT-OSS TE model on one H100 in nvcr.io/nvidian/nemo:26.08.rc4

The checkpoint plan now fails on optional empty TE core-attention state:

RuntimeError: Missing key in checkpoint state_dict: decoder.layers.self_attention.core_attention._extra_state/shard_0_24.

Job 15111919 failed 1:0 after 4m14s. It created no HF output; both public source trees remain clean. This is not a verification pass, and no local modification will be used to make one. The owning MCore fix will be public with focused regression coverage before another clean pushed full-workflow rerun.

[cuichenx]

Advanced this integration PR normally to signed/DCO head 99433894a7c6ec98a25079e123038b60cdff53c3, pinning exact public MCore PR #6270 head d57d882fd08ee1762eab6144260cf9693f58ac9d. No public history was rewritten.

• exact-public MCore job 15112718: COMPLETED 0:0; all eight H100 ranks reported all five dense/MoE checkpoint-key and TE norm/attention extra-state cases passed
• Bridge conversion, GPT-provider, and disabled-RNG checkpoint tests: 38 passed in driverless nvcr.io/nvidia/nemo:26.06.01
• uv lock --check: passed
• uv run --no-sync pre-commit run --all-files: passed
• GitHub commit signature: valid

The CPU checkpoint from Bridge dbe0a5b7... / MCore 9ba6247b... will not be reused as final proof for this new dependency head. I will generate a new full GPT-OSS 20B checkpoint from a fresh clean public checkout of 99433894..., then load/export it with the real H100/TE model. Until both complete, the workflow remains not verified.

[cuichenx]

/ok to test 9943389

[cuichenx]

Fresh exact-public full GPT-OSS 20B CPU-to-H100 validation exposed another strict export defect after clearing all previously reported checkpoint-load blockers.

• Bridge head: 99433894a7c6ec98a25079e123038b60cdff53c3
• MCore head: d57d882fd08ee1762eab6144260cf9693f58ac9d
• CPU import: zero-GPU job 15113186 completed 0:0, loaded all 20,914,757,184 parameters, and produced a complete fresh checkpoint
• H100/TE load/export: job 15114263 on one H100 in nvcr.io/nvidian/nemo:26.08.rc4
• The full TE model loaded the CPU checkpoint far enough to clear the prior MoE norm key, TE norm extra-state, and TE core-attention extra-state failures.
• Strict HF export then failed with:

KeyError: "Tensor 'model.layers.0.mlp.experts.gate_up_proj' from generator not found in the original model structure. Re-run with strict=False to save the partial checkpoint instead of failing."

I will not use strict=False or count a partial export. This is a failed exact-public candidate validation, not verification. The owning Bridge GPT-OSS/HF export mapping must be fixed in a public commit with focused regression coverage, then the original clean full workflow must be rerun.

[cuichenx]

/ok to test d3346db

[cuichenx]

Published signed candidate d3346db by normal fast-forward push. This preserves strict safetensors key accounting while allowing GPT-OSS dequantized gate_up_proj and down_proj tensors to replace their physical MXFP4 block and scale pairs in the output inventory; replaced keys must exist and share one source shard. Both ordinary and distributed saver paths use the transformed map. All-file pre-commit passed, and 34 focused unit cases passed against the mounted candidate checkout in nvcr.io/nvidia/nemo:26.06.01. Independent exact-diff review found no blockers. This is candidate evidence only, not model verification. A fresh zero-GPU import and H100/TE strict export from exact clean public heads is still required.

[cuichenx]

Exact-public full GPT-OSS CPU-to-H100 conversion gate passed at Bridge d3346db / MCore d57d882fd08ee1762eab6144260cf9693f58ac9d. Fresh zero-GPU job 15114882 completed 0:0 on cpu1-00035 in 18m19s in nvcr.io/nvidia/nemo:26.06.01, loaded all 20,914,757,184 parameters, and wrote a new 41,839,125,360-byte distcp shard. Mandatory job 15115717 then completed 0:0 on one H100 at pool0-01779 in 5m44s in nvcr.io/nvidian/nemo:26.08.rc4, constructed the full TE model, loaded only that new checkpoint, converted all 3,291 bridge tasks, and completed strict distributed HF export. The output index has 411 keys across 3 shards, contains the dequantized expert gate_up_proj/down_proj keys, contains no expert MXFP4 _blocks/_scales keys, and every referenced shard exists. Bridge and MCore heads still matched their public refs with zero status entries after both runs. This validates the full CPU import plus H100/TE strict load/export gate; forward/numerical parity is not claimed here.