Support Qwen3next - #939
Conversation
| @@ -2048,7 +1876,7 @@ def _build_sharded_state_dict_metadata( | |||
| else: | |||
| metadata["distrib_optim_sharding_type"] = "dp_zero_gather_scatter" | |||
| metadata["chained_optim_avoid_prefix"] = True | |||
| metadata["singleton_local_shards"] = False | |||
| metadata["singleton_local_shards"] = True | |||
There was a problem hiding this comment.
why is this change needed?
There was a problem hiding this comment.
According to this line of code in megatron-lm, singleton_local_shards is set to False if and only if args.dist_ckpt_save_pre_mcore_014 or force_pre_mcore_014.
Honestly, I don't fully understand this flag, but in my actual runs, I've found that setting metadata["singleton_local_shards"] = False causes the tensor parallel sharding to be incorrect in MCore.
| "output_layer.weight": "lm_head.weight", | ||
| "decoder.final_layernorm.weight": "model.norm.weight", | ||
| } | ||
| for megatron_param, hf_param in layer_mappings.items(): |
There was a problem hiding this comment.
can we just write down the mapping directly, not use layer_mappings. It's a bit inconsistent with other model.
We feel it is clearer just write the whole name together, though we need to duplicate the prefix.
There was a problem hiding this comment.
Yes. I will do the change later.
There was a problem hiding this comment.
Thanks. Changes done. Does it look good now?
|
Hi, thanks for contribution. unit tests/ functional tests need to be added. also it will be nice if you can attach your validation results. |
1779982 to
f388215
Compare
839d368 to
27c25d4
Compare
|
moved the PR to #1068 |
2 participants
No description provided.