Skip to content

fix(conversion): initialize CPU imports with Gloo - #5313

Closed
cuichenx wants to merge 2 commits into
mainfrom
agent/cpu-import-gloo-verification-gate
Closed

fix(conversion): initialize CPU imports with Gloo#5313
cuichenx wants to merge 2 commits into
mainfrom
agent/cpu-import-gloo-verification-gate

Conversation

@cuichenx

@cuichenx cuichenx commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What changed

  • run CPU Hugging Face to Megatron import construction and checkpoint save inside a temporary single-rank Gloo context
  • preserve a caller-managed distributed context when one already exists
  • add focused regression coverage for both paths
  • update the model-verification-card skill to reject local patches, unpublished commits, source overlays, monkeypatches, and copied trees as verification evidence

Root cause

AutoBridge.import_ckpt() requested CPU model initialization without first initializing a CPU process group. provide_distributed_model() therefore entered its standalone fallback, unconditionally selected a CUDA device, and initialized NCCL. A genuine CPU-only import failed before weight mapping.

User impact

CPU checkpoint import can construct and save through the existing temporary Gloo helper without requiring the CUDA/NCCL fallback. Verification failures must now be fixed through a pushed upstream PR and rerun from that exact clean commit before a card item can be marked verified.

Validation

  • uv run --no-project python -m pytest tests/unit_tests/models/test_auto_bridge.py -k "import_ckpt" tests/unit_tests/training/test_model_load_save.py -k "import_ckpt or temporary_distributed_context" -q — 7 passed
  • uv run --no-sync pre-commit run --all-files — passed
  • skill quick_validate.py — passed
  • git diff --check — passed

Pending model validation

The full GPT-OSS CPU import/export rerun will use the exact pushed PR head. It remains pending and is not claimed as verified here. The newer Transformer Engine load-time CUDA-driver dependency is a separate upstream blocker for driverless 26.08 containers.

Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@cuichenx cuichenx added bug Something isn't working area:ckpt Checkpoint conversion, loading, export, and save paths labels Aug 4, 2026
@cuichenx

cuichenx commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Full openai/gpt-oss-20b CPU import from this exact pushed PR head was run against the pinned MCore commit. This PR cleared the original unconditional CUDA-device/NCCL fallback and reached model construction. The next failure is an independent MCore bug: YARN constructor-time CPU initialization calls torch.cuda.current_device().

That owning fix is now public as NVIDIA/Megatron-LM#6270. Its focused eight-rank tests pass from the exact pushed MCore PR head. End-to-end GPT-OSS CPU import remains pending and is not verified; it must be rerun from a clean pushed Bridge commit that consumes the public MCore fix.

@cuichenx

cuichenx commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b73e19e

@cuichenx

Copy link
Copy Markdown
Contributor Author

Closing as superseded. Current main already contains equivalent temporary Gloo scoping for CPU import through #5007. The broader CPU-only conversion gaps are being tracked separately.

@cuichenx cuichenx closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ckpt Checkpoint conversion, loading, export, and save paths bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant