fix: use explicit Megatron-FSDP V1/V2 classes in isinstance checks - #5357
Closed
shanmugamr1992 wants to merge 1 commit into
Closed
fix: use explicit Megatron-FSDP V1/V2 classes in isinstance checks#5357shanmugamr1992 wants to merge 1 commit into
shanmugamr1992 wants to merge 1 commit into
Conversation
Signed-off-by: shanmugamr1992 <shanmugamr1992@gmail.com>
This was referenced Aug 5, 2026
Contributor
|
Superseded by merged #5431. Current |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by the nemo-rl-testing-agent while validating
NVIDIA/Megatron-LM#5382
against the NeMo-RL Megatron functional suites. The break is not that PR's
fault — it reproduces on megatron-core
main, so every NeMo-RL Megatron L1test is currently failing at import time.
Failure
All 7 tests in
L1_Functional_Tests_Megatron_4failed with an identicalsignature, reached via
build_conversion_tasks->_get_pg_collection_from_model->unwrap_model:Root cause
Megatron-LM #5865 split the
FSDP adapter into explicit
FullyShardedDataParallelV1/FullyShardedDataParallelV2classes and turned
FullyShardedDataParallelinto a factory function thatdispatches between them. The old name is therefore no longer a type, and any
isinstance()against it raises. The factory's own docstring says so:Bridge has three
isinstancesites against that name, in three different states:mainmodels/conversion/utils.py(unwrap_model)training/train.py(FSDP manual buffer registration)training/setup.pyFix
Type-check against the explicit classes at the two crash sites, using the same
try/except ImportErroridiomtraining/setup.pyalready uses so oldermegatron-core (where
FullyShardedDataParallelis still a class) keeps working.V1 and V2 were both introduced by #5865, so the two-name import succeeds or
fails together.
One thing left for a maintainer rather than folded in here:
training/setup.pyimports
FullyShardedDataParallelV1only, so itsisinstancecheck silentlymisses a V2-wrapped model instead of crashing. That looked like a behavioural
question the functional suite does not exercise, so I did not touch it.
Validation
L1_Functional_Tests_Megatron_4on oci-hsg (GB200, 4 GPUs), NeMo-RL86774472, megatron-core at Megatron-LM#5382 headeb01b689:TypeErrorabove.grpo_megatron_generation_topp_topk, no longercrashes — it runs to completion and misses a metric threshold
(
max(train/token_mult_prob_error)1.078 vs< 1.06). That is a separatequestion being baselined against megatron-core
mainindependently, and isunrelated to this change.
Draft: raised by an agent, needs a human review before merge.