examples/multimodal_dev: docs + Victarry review fixes + vision patch merger parity test

[wplf]

Cherry-picked from feat/qwen35-vl-example onto
jinliangl/qwen35-vl-hybridep-deploy-pr4715 so the review diff is scoped
to just three commits of examples/multimodal_dev/ work:

Commit	Summary
075fe8935	docs(examples/multimodal_dev): add checkpoint conversion guide to README — new ## Checkpoint Conversion (HF → Megatron-FSDP DTensor) section (Setup / Convert / Output / Bridge dependency)
4e853ea47	fix(examples/multimodal_dev): address Victarry review feedback — resolves the 6 inline review comments on upstream PR NVIDIA#4751: PatchMerger GELU tanh → none (HF parity), PP>1 fail-fast in pretrain_multimodal.py, EP default 2 → 1 + NUM_EXPERTS=0 && EP>1 fail-fast in run_qwen35_vl.sh, --moe-router-force-load-balancing gated behind FORCE_LOAD_BALANCING=1, vlm_dataset.py → cord_v2.py rename + registry path update + extensibility docstring, MRoPE precompute TODO note
2130f4210	test(examples/multimodal_dev): add Qwen35VLPatchMerger HF parity test — new tests/test_vision_patch_merger_parity.py. Inlines HF Qwen3VLVisionPatchMerger, copies state-dict 1:1 (TP=1), asserts logits parity. Runs on H100: fp32 max-abs 2.55e-5 (atol=1e-4), bf16 max-abs 3.91e-3 (atol=5e-2); test passes

Total: 9 files / +325 -10 (incl. 1 rename).

Verified on cw: parity test job 12224061 COMPLETED 0:0 in 43s.

Summary by Sourcery

Document checkpoint conversion from HuggingFace to Megatron-FSDP DTensor, tighten multimodal training/launch configuration safety, and add a numerical parity test for the Qwen35VLPatchMerger vision module against the HuggingFace reference.

Enhancements:

• Disallow pipeline parallelism in multimodal_dev by failing fast when pipeline_model_parallel_size > 1 to prevent unsupported configurations.
• Clarify the CORD-V2 dataset module naming and description to better reflect its role as the reference CORD-V2 VLM dataset.
• Align the Qwen35VLPatchMerger GELU activation with the HuggingFace Qwen3VLVisionPatchMerger behavior for numerical consistency.
• Document a performance note on MRoPE position metadata computation to guide future optimization of CUDA graph compatibility.

Documentation:

• Add a checkpoint conversion guide for importing HuggingFace Qwen3.5 models into Megatron-FSDP DTensor via Megatron-Bridge, including setup, invocation, and dependency notes.

Tests:

• Introduce a distributed parity test that compares Qwen35VLPatchMerger outputs to an inlined HuggingFace reference implementation in fp32 and bf16 with strict numerical tolerances.

Chores:

• Update the Qwen35-VL example launch script to default expert parallelism to 1, gate forced MoE router load-balancing behind an explicit opt-in flag, and fail fast on invalid dense-model expert-parallel configurations.
• Switch the dataset registry entry from the generic VLM dataset module to the renamed CORD-V2-specific implementation.

[sourcery-ai]

Reviewer's Guide

Adds HF→Megatron-FSDP DTensor checkpoint conversion docs for multimodal_dev, tightens configuration safety/semantics for Qwen3.5-VL scripts and model (PP/EP/MoE, dataset registry, GELU parity, MRoPE perf note), and introduces a numerical parity test that validates Qwen35VLPatchMerger against the HuggingFace reference implementation.

Sequence diagram for Qwen35VLPatchMerger HF parity test

sequenceDiagram
    participant PyTest as PyTest
    participant HFMerger as Qwen3VLVisionPatchMerger
    participant LocalMerger as Qwen35VLPatchMerger

    PyTest->>HFMerger: load_state_dict(hf_state_dict)
    PyTest->>LocalMerger: load_state_dict(hf_state_dict)

    PyTest->>HFMerger: forward(hidden_states)
    HFMerger-->>PyTest: hf_logits

    PyTest->>LocalMerger: forward(hidden_states)
    LocalMerger-->>PyTest: local_logits

    PyTest->>PyTest: assert torch.allclose(hf_logits, local_logits, atol)

Loading

File-Level Changes

Change	Details	Files
Document HF → Megatron-FSDP DTensor checkpoint conversion flow for Qwen3.5-VL multimodal_dev.	Add a README section describing how to clone/pin Megatron-Bridge against this branch and run convert_checkpoints_fsdp.py to import HF Qwen3.5 checkpoints into Megatron-FSDP DTensor format. Document output directory layout for converted checkpoints and constraints on TP/CP/EP topology. Call out dependency on a specific Megatron-Bridge PR to avoid tokenizer-related AttributeError after save.	examples/multimodal_dev/README.md
Harden Qwen3.5-VL training script configuration for MoE/EP and gate router force-load-balancing behind an explicit opt-in flag.	Clarify that pipeline parallelism (PP) must stay 1 for multimodal_dev in script docs and default EP to 1 instead of 2, with MoE variants overriding EP as needed. Introduce FORCE_LOAD_BALANCING env var to optionally enable --moe-router-force-load-balancing only for perf/mock-data runs, and remove the unconditional flag from the common GPT_MODEL_ARGS. Add a fail-fast check that disallows EP>1 when NUM_EXPERTS=0 (dense variants) to avoid invalid MoE arg wiring.	examples/multimodal_dev/scripts/run_qwen35_vl.sh
Enforce at runtime that multimodal_dev does not run with pipeline_model_parallel_size > 1.	Add a guard in pretrain_multimodal.py that raises ValueError when pipeline_model_parallel_size > 1, explaining that the model provider ignores PP stage flags and builds the full model on every rank.	examples/multimodal_dev/pretrain_multimodal.py
Refine and clarify the CORD-V2 multimodal dataset module and its registry entry.	Rename vlm_dataset.py to cord_v2.py and update the dataset registry path to point at the new module name. Update the dataset module docstring to position it explicitly as the reference implementation for the CORD-V2 receipt OCR dataset and describe its scope/limitations.	examples/multimodal_dev/data/vlm_dataset.py examples/multimodal_dev/data/cord_v2.py examples/multimodal_dev/models/__init__.py
Add a performance-oriented TODO note for Qwen3.5-VL MRoPE position metadata computation.	Extend the mrope._build_sample_mrope_positions docstring to call out GPU-CPU syncs caused by .tolist()/.item(), their impact on CUDA graph capture, and suggest moving/caching the computation off the hot path in a follow-up.	examples/multimodal_dev/models/qwen35_vl/mrope.py
Match HuggingFace Qwen3VLVisionPatchMerger activation behavior for vision patch merger.	Change the GELU activation in Qwen35VLPatchMerger from approximate='tanh' to approximate='none' to align numerically with the HuggingFace reference implementation, documenting this in a comment.	examples/multimodal_dev/models/qwen35_vl/vision_encoder.py
Introduce a HuggingFace parity test for Qwen35VLPatchMerger covering fp32 and bf16.	Add a standalone distributed test script that inlines the HF Qwen3VLVisionPatchMerger module, builds a matching Qwen35VLPatchMerger under Megatron core, and copies weights 1:1 in TP=1. Configure a minimal TransformerConfig and Megatron parallel state (TP=PP=1), run both modules on shared random inputs in fp32 and bf16, and assert close with tight atol/rtol thresholds while printing max/mean abs diffs. Ensure the test is runnable via torchrun without requiring transformers as a dependency by inlining the HF reference and managing sys.path to include the repo root.	examples/multimodal_dev/tests/test_vision_patch_merger_parity.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In the checkpoint conversion README snippet, the Git instructions are pinned to a personal fork/feature branch (wplf feat/qwen35-vl-example); consider referencing an upstream branch or specific commit hash (and/or calling out that this is temporary) so the instructions remain valid as your personal branches evolve.
• The parity test script mutates sys.path to inject the repo root; consider relying on package-relative imports (or running via python -m ...) instead of manual sys.path surgery to make the test more robust to different execution environments.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In the checkpoint conversion README snippet, the Git instructions are pinned to a personal fork/feature branch (`wplf feat/qwen35-vl-example`); consider referencing an upstream branch or specific commit hash (and/or calling out that this is temporary) so the instructions remain valid as your personal branches evolve.
- The parity test script mutates `sys.path` to inject the repo root; consider relying on package-relative imports (or running via `python -m ...`) instead of manual `sys.path` surgery to make the test more robust to different execution environments.

## Individual Comments

### Comment 1
<location path="examples/multimodal_dev/tests/test_vision_patch_merger_parity.py" line_range="57-60" />
<code_context>
+HIDDEN_SIZE = 1152
+OUT_HIDDEN_SIZE = 3584
+SPATIAL_MERGE_SIZE = 2
+NUM_PATCHES = 64  # must be divisible by spatial_merge_size ** 2
+
+ATOL_FP32 = 1e-4
</code_context>
<issue_to_address>
**suggestion (testing):** Turn the implicit divisibility requirement into an assertion to guard future changes.

The test currently only documents that `NUM_PATCHES` must be divisible by `SPATIAL_MERGE_SIZE ** 2` but doesn’t enforce it. Please add an assertion like `assert NUM_PATCHES % (SPATIAL_MERGE_SIZE ** 2) == 0` so future constant changes that violate this invariant fail clearly rather than causing confusing layout/parity issues.

```suggestion
SPATIAL_MERGE_SIZE = 2
NUM_PATCHES = 64  # must be divisible by spatial_merge_size ** 2
assert NUM_PATCHES % (SPATIAL_MERGE_SIZE**2) == 0

ATOL_FP32 = 1e-4
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

suggestion (testing): Turn the implicit divisibility requirement into an assertion to guard future changes.

The test currently only documents that NUM_PATCHES must be divisible by SPATIAL_MERGE_SIZE ** 2 but doesn’t enforce it. Please add an assertion like assert NUM_PATCHES % (SPATIAL_MERGE_SIZE ** 2) == 0 so future constant changes that violate this invariant fail clearly rather than causing confusing layout/parity issues.

Suggested change

	SPATIAL_MERGE_SIZE = 2
	NUM_PATCHES = 64 # must be divisible by spatial_merge_size ** 2
	ATOL_FP32 = 1e-4
	SPATIAL_MERGE_SIZE = 2
	NUM_PATCHES = 64 # must be divisible by spatial_merge_size ** 2
	assert NUM_PATCHES % (SPATIAL_MERGE_SIZE**2) == 0
	ATOL_FP32 = 1e-4