[model] feat: Sync qwen_vl tree with upstream main, keep CP pre-shard fix - #19
Merged
Conversation
… fix Sync src/megatron/bridge/models/qwen_vl/ (13 files) and its unit tests (7 files) to NVIDIA-NeMo/Megatron-Bridge main (bfc4b41): the Qwen3.5-VL bridge now derives from the text-only Qwen35(MoE)Bridge, and the modelling_qwen3_vl stack picks up upstream's restructured forward with its own packed-CP partition path. Additive-only changes to shared files (existing code untouched): - packed_seq_utils.py: add get_packed_seq_q_cu_seqlens, get_packed_seq_cp_partition_indices, unpack_mcore_thd_tensor_for_ position_ids, repack_mcore_thd_position_ids (upstream versions); keep this branch's get_packed_seq_params as-is - flop_utils.py: add accumulate_flops_metadata and its private helpers Re-apply the local CP fix eeb84d7 onto the restructured model.py: _cp_local_vision_embed_indices verbatim, AllGatherVisionEmbeddings.apply positional args, and a new packed_input_pre_sharded gate so upstream's partition path does not double-slice inputs a framework already sharded across CP ranks (cu_seqlens_q[-1] == cp_size * len(input_ids)). Deliberately NOT synced: qwen_vl/data/ (new upstream energon collate stack; depends on data/sequence_batching.py, data/vlm_processing.py and a GenericVisualInputs field this branch does not have), recipes, and the shared conversion files (param_mapping/peft_bridge/model_bridge). Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Sync the Qwen VL model tree with NVIDIA-NeMo/Megatron-Bridge
main(bfc4b418), following up on #18:models/qwen_vl/(13 files):qwen35_vl_bridge.pynow derives from the text-onlyQwen35(MoE)Bridge(from [model] feat: Add text-only Qwen3.5 bridges (port from upstream main) #18);modelling_qwen3_vl/*picks up upstream's restructured forward (packed-CP partition path, MTP spec builders, rope/text-model updates)tests/unit_tests/models/qwen_vl/(7 files) synced to matchpacked_seq_utils.py,accumulate_flops_metadata(+private helpers) intoflop_utils.pyeeb84d7f(_cp_local_vision_embed_indices+ positionalAllGatherVisionEmbeddings.apply), re-applied onto the restructuredmodel.pywith a newpacked_input_pre_shardedgate: upstream's new partition path must not double-slice inputs that a training framework (miles) already pre-sharded across CP ranks (cu_seqlens_q[-1] == cp_size * len(input_ids))Deliberately NOT synced
qwen_vl/data/(new upstream energon collate stack) — depends ondata/sequence_batching.py,data/vlm_processing.py, and aGenericVisualInputsfield this branch does not have; nothing here references itrecipes/and shared conversion files (param_mapping.py,peft_bridge.py,model_bridge.py, bulk offlop_utils.py) — out of scope; audited that the synced tree only needs symbols that already exist hereGLM5 impact: none (verified)
models/glm5/andmodels/qwen_vl/(both directions)Validation
ruff checkclean on all changed filespytest tests/unit_tests/models/{qwen_vl,qwen,glm,glm_vl}— 351 passed, 2 skippedpp_rank=kwarg on the experimental-attention spec,scatter_to_sequence_parallel_region(group=),get_tensor_model_parallel_group_if_none— all present