fix(conversion): initialize CPU imports with Gloo - #5313
Conversation
Signed-off-by: Chen Cui <chcui@nvidia.com>
Signed-off-by: Chen Cui <chcui@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Full That owning fix is now public as NVIDIA/Megatron-LM#6270. Its focused eight-rank tests pass from the exact pushed MCore PR head. End-to-end GPT-OSS CPU import remains pending and is not verified; it must be rerun from a clean pushed Bridge commit that consumes the public MCore fix. |
|
/ok to test b73e19e |
|
Closing as superseded. Current main already contains equivalent temporary Gloo scoping for CPU import through #5007. The broader CPU-only conversion gaps are being tracked separately. |
What changed
Root cause
AutoBridge.import_ckpt()requested CPU model initialization without first initializing a CPU process group.provide_distributed_model()therefore entered its standalone fallback, unconditionally selected a CUDA device, and initialized NCCL. A genuine CPU-only import failed before weight mapping.User impact
CPU checkpoint import can construct and save through the existing temporary Gloo helper without requiring the CUDA/NCCL fallback. Verification failures must now be fixed through a pushed upstream PR and rerun from that exact clean commit before a card item can be marked verified.
Validation
uv run --no-project python -m pytest tests/unit_tests/models/test_auto_bridge.py -k "import_ckpt" tests/unit_tests/training/test_model_load_save.py -k "import_ckpt or temporary_distributed_context" -q— 7 passeduv run --no-sync pre-commit run --all-files— passedquick_validate.py— passedgit diff --check— passedPending model validation
The full GPT-OSS CPU import/export rerun will use the exact pushed PR head. It remains pending and is not claimed as verified here. The newer Transformer Engine load-time CUDA-driver dependency is a separate upstream blocker for driverless 26.08 containers.