feat(grpo): deduplicate multimodal payloads by aroshanghias-nvd · Pull Request #3527 · NVIDIA-NeMo/RL

aroshanghias-nvd · 2026-08-06T18:30:46Z

Deduplicate multimodal GRPO payloads

What this changes

Represents repeated multimodal rows as logical row-to-segment mappings over a
compact physical tensor store, while retaining normal logical batch semantics.
Preserves media identity through GRPO G-expansion, slicing, concatenation,
dynamic batching, sequence packing, DP sharding, async replay, DAPO assembly,
and replay-buffer checkpoint restore.
Re-interns equivalent physical media after shard formation so DP-local Ray
transfers retain savings even when packing or sharding splits prompt groups.
Covers image, video, and audio through shared primitives. Model-level
qualification is currently Nemotron image and Qwen2.5-VL image; audio/video
behavior is covered by focused data-structure tests rather than model runs.
Supports native NeMo-RL rollout and NeMo Gym rollout, sync and async GRPO.
Gym request payloads are not expected to shrink because Gym receives each
request independently; Gym return, replay, and policy-training boundaries do.
Independently fixes native multi-turn VLM rollout generation to stop reusing
stale turn-0 vllm_content; later turns use the current conversation tokens and
media. This behavior correction is unconditional, not gated by deduplication.
Adds opt-in exact Ray-boundary payload metrics and forwards them to W&B.
Pre-casts policy-bound pixel_values from fp32 to the configured policy
precision before Ray transfer (bf16 in the shipped recipes), matching Omni MR
!12 and halving physical pixel-tensor bytes for bf16. Preprocessing, Gym return,
and replay storage retain fp32; both dedup modes use the same cast.
Keeps shared VLM defaults off. The maintained Nemotron VLM/Omni recipes enable
deduplication; payload metrics remain debug-only and disabled in recipes.

Representation and safety contract

PackedTensor stores physical segments plus CSR-like row offsets and segment
indices. Each physical segment also carries opaque provenance.
Provenance identity is the sole evidence used to re-intern physical segments;
tensor values are not compared. Value-preserving copies and device moves retain
provenance, while value-changing dtype conversion issues fresh provenance. Missing media rows are explicit,
and correlated media keys preserve per-row logical counts and ordering.

Flag-off retains the legacy expanded logical representation. Both modes use the
same configured pixel_values dtype for policy-bound transfers, so dedup A/B
runs do not differ in dtype. Legacy pickles and checkpoints remain readable.
Compact replay checkpoints also remain readable with the flag off because the
compact representation is self-describing inside each saved message log.

Quantified G=16 evidence

The W&B campaigns below quantify deduplication before the bf16 pre-cast was
ported. They remain the structural off/on evidence. The pre-cast applies equally
to both modes and independently halves physical policy-bound pixel_values
bytes; the focused protocol-5 test above validates the current combined path.

Nemotron Omni 30B, Megatron, async NeMo Gym

4 nodes, 8 GPUs/node, TP=2, EP=16, CP=2, 10 steps. Dedup off/on jobs
14856877/14856876 completed with exact logical trajectory, reward, and token
parity. TMPE mean/max was 1.022114/1.028493 off and
1.021894/1.027261 on.

Exact Ray boundary	Serialized off/on reduction
Gym request	1.00x (expected: requests are independent)
Gym return	284.60x
Replay push	15.49x
Replay sample	15.55x
Rollout to policy	14.90x
Policy logprobs	3.94x
Reference logprobs	3.93x
Policy train	3.91x

W&B: off,
on.

An independently repeated one-step final-telemetry pair (14869430/14869431)
matched all 16 logical rows, rewards, token counts, masks, advantages, content,
and agent references. TMPE was 1.014925 off and 1.014564 on. Its reductions
were 620.02x at Gym return, 15.78x at replay push/sample, 13.41x rollout-to-policy,
and 3.77-3.85x across policy shards.

Qwen2.5-VL 3B, Megatron, native rollout

1 node, TP=1, DP=8, G=16, 10 steps. Jobs 14858320/14858321 matched logical
trajectories, rewards, and generated tokens exactly. Mean loss was
0.0748708278 off and 0.0748722717 on; maximum paired loss delta was
1.21668e-5. TMPE mean/max was 1.022007/1.025749 off and
1.021798/1.024828 on. Serialized reductions were 14.97x at prompt repeat,
17.05x at vLLM generation, 15.06x rollout-to-policy, and approximately 1.99x
across policy shards.

W&B: off,
on.

Qwen2.5-VL 3B, AutoModel, native rollout

1 node, DP=8, G=16, 10 sampled steps. Jobs 14979492/14979493 completed all
steps. The first complete generation-to-training step matched exactly; later
independently updated sampled trajectories diverged, so this qualifies the
AutoModel data path and multi-step health, not strict multi-step A/B identity.
Serialized reductions were 14.96x prompt repeat, 17.05x vLLM generation,
15.29x rollout-to-policy, and approximately 1.99x at policy boundaries.

W&B: off,
on.

Gemma 3 4B, AutoModel, transport-only diagnostic

A 10-step native-rollout G=16 pair on pre-rebase candidate 5496708e also
exercised the generic AutoModel transport path. The first-step trajectory hash
matched exactly. Averaged across ten steps, serialized payloads fell 43.12x at
the exact sharded vLLM-generation Ray boundary, 15.68x in the rollout-to-policy
batch, and approximately 2.00x at the sharded policy logprob, reference-logprob,
and training Ray boundaries. The 2x shard-local result is expected because 16
logical generations were distributed over eight AutoModel DP workers.

This is transport-only evidence. Gemma is not included in model correctness or
performance qualification because its unrelated vLLM/AutoModel token-logprob
mismatch was catastrophic with deduplication both disabled and enabled.

W&B: off,
on.

Current-#3414 validation

Implementation-equivalent full unit campaign: 956 passed, 6 skipped
(SLURM 15214209). This included full GRPO, PPO, distillation, Gym/replay,
batching, AutoModel data, vLLM utils, config validation, serialization, and
payload-metric suites.
Megatron CP=2 model tests: 2 passed (SLURM 15214753), including model-ingress,
logprob/loss/gradient parity and distributed-checkpoint round-trip.
The dedup patch was replayed without conflicts from c3b8dacb onto the latest
feat: Multimodal nemo gym compatible grpo pipeline #3414 head f27dc6d6; git range-diff confirms the product patch is identical.
The intervening feat: Multimodal nemo gym compatible grpo pipeline #3414 delta only changes PPO documentation/configuration,
ppo.py, and dataset utilities, with no file overlap with this patch.
On implementation commit 91959244, Ruff format/check passed and Pyrefly
reported zero shown errors across 219 modules (255 pre-existing ignored
diagnostics) in SLURM 15222427. The focused
GRPO/PPO/data/Gym/replay/backend/config/payload suite passed on GPU with
1004 passed, 6 skipped in SLURM 15222850.
Bf16 policy-transport commit d95bb1ae ports Omni MR !12 to GRPO without
materializing deduplicated rows. Focused validation: 196 generic GRPO/data/
payload tests passed (SLURM 15234623), 39 AutoModel data tests passed
(15234855), and 41 Megatron data tests passed with one expected 2-GPU skip
(15234856). Ruff format/check passed; Pyrefly reported zero errors on the
changed whitelisted module. The payload test verifies exactly 2x fewer
physical pixel_values bytes plus lower protocol-5 serialized size.
Scope-cleanup commit 29de7b52 removes the standalone design document and
redundant per-recipe debug defaults. The resulting complete configuration set
passed with 547 passed, 6 skipped in SLURM 15224236.
An independent review of the implementation-equivalent c3b8dacb patch found
no code-level merge blockers.
Implementation-equivalent Qwen2.5-VL 3B AutoModel native G=16 jobs
15216656/15217828 completed two steps each. The first trajectory hash
matched exactly; later independently sampled trajectories diverged after
optimizer/refit, as expected for this non-frozen smoke. Aggregate serialized
reductions were 14.99x prompt repeat, 17.06x vLLM generation, 15.30x
rollout-to-policy, 1.994x policy/reference logprobs, and 1.991x policy train.
W&B: off,
on.
Implementation-equivalent Nemotron Omni 30B Megatron async Gym G=16, TP=2,
EP=16, CP=2 job 15219896 completed two steps per side on four 8-GPU nodes. All
losses were finite and TMPE stayed healthy: 1.01897-1.01950 off and
1.02197-1.02418 on. Aggregate serialized reductions were 406.68x at Gym
return, 15.67x replay push, 15.69x replay sample, 15.28x rollout-to-policy,
3.97x policy logprobs, and 3.96x policy train; Gym request was exactly 1.00x.
These were independent sampled runs, so the frozen 10-step campaign above,
rather than this smoke, supplies the strict parity claim. W&B:
off,
on.

Evaluation limits

Gemma 3 is excluded from correctness and performance qualification: its
vLLM/AutoModel token-logprob mismatch is catastrophic with dedup both off and
on. Its transport-only diagnostic is reported separately above.
Audio/video model runs are not claimed; shared media primitives have focused
coverage and are intentionally media-type agnostic.
AutoModel VLM CP>1 is rejected by the upstream worker independently of this
change. Megatron CP=2 is qualified; CP>2 is not claimed.
Payload instrumentation is opt-in because protocol-5 serialization is itself
nontrivial work. It is intended for qualification/debugging, not production.

Test plan

PackedTensor logical/physical invariants, copy, pickle, legacy restore,
empty rows, missing keys, cross-key alignment, and differing equivalence groups.
Repeat/slice/concat/from-batches/dynamic-batch/sequence-pack/DP-shard
normalization and shard-local re-interning.
Native and Gym sync/async integrations, multi-turn stale-media handling,
replay/DAPO/checkpoint restore, and flag-off behavior.
Non-GRPO primitive blast radius: full PPO and distillation unit suites.
AutoModel CP=1 worker materialization and Megatron CP=2 model tests.
Exact sharded Ray-argument telemetry and W&B aggregation.
Protected CI after PR submission.

…mbing and chat_template parity Folds three WIP commits (b16fca2 + 5486b57 + cd349d0): - Add tokenizer_config field to NemoGymConfig and processor reconstruction inside the actor for multi-turn multimodal postprocessing. - Add multimodal utilities (encode_images_in_examples, extract_multimodal_model_inputs, process_multimodal_chat, resolve_to_image) and their consumers in NemoGym. - Add sync single-step polish across nemo_gym.py / multimodal_utils.py / processors.py. - Maintain chat_template kwargs parity between the async vLLM worker and HF. Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 3215893) Signed-off-by: rohitrango <rohit.rango@gmail.com>

…ntroller pathway Reword of aa04447 (async plumbing). Adds the wiring in grpo.py and run_multimodal_grpo_nemo_gym.py so that multimodal NeMo-Gym rollouts flow through the async single-controller path introduced in the multimodal foundation commit. Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit c755a25) Signed-off-by: rohitrango <rohit.rango@gmail.com>

…ygon-naming + multi-turn tool calling) Folds four WIP commits (28ba38e + 81f885f + ecf72e4 + 7171787): - Bump seq len and generation count on the doorkey smoke config. - Add tangram env smoke config and its launcher. - Add multi-turn multimodal tool calling / polygon-naming smoke config and launcher. Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit b54e6a8) Signed-off-by: rohitrango <rohit.rango@gmail.com>

…m_to_nemo_rl_result Commit ec8333d added `nemo_gym_row: dict` as a new positional parameter to `NemoGym._postprocess_nemo_gym_to_nemo_rl_result` but never referenced it in the function body, and neither the sole in-tree caller (`_run_rollouts_iterator` in the same file) nor the six unit-test call sites (`tests/unit/environments/test_nemo_gym{,_router_replay}.py`) were updated to pass it. As a result every sync/async multimodal smoke recipe under `examples/nemo_gym/run_gymv_smoke.sh` crashed with TypeError: NemoGym._postprocess_nemo_gym_to_nemo_rl_result() missing 1 required positional argument: 'tokenizer' as soon as the first rollout came back from the Gym HTTP server — the `nemo_gym_result` positional was being consumed by the phantom `nemo_gym_row` slot, so `tokenizer` looked missing. Since the parameter is unused, the minimal fix is to remove it and restore the original 2-arg `(nemo_gym_result, tokenizer)` signature. This lines back up with all six test call sites (which were already passing 2 args) and the production caller at nemo_gym.py:442 (which was already passing 2 args), so no other files need touching. Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit a2f8fd8) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 7ddf223) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 7bf55af) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit b7fc746) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com>

except assistant) Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 85a24b8) Signed-off-by: rohitrango <rohit.rango@gmail.com>

`_index_per_turn_images` gated its bucket flush on `role == "assistant"`, but `_postprocess_nemo_gym_to_nemo_rl_result` treats every item carrying truthy `generation_token_ids` as a trainable turn — including reasoning-only responses and `function_call` items whose role is not `"assistant"`. The mismatch left the batched flatten path with a `PackedTensor` for normal assistant turns and a missing entry for reasoning/tool-call turns, crashing `PackedTensor.flattened_concat` on async multimodal GRPO runs. Gate the flush on `generation_token_ids` directly so the per-turn image list stays aligned with the postprocess loop's `turn_idx`, and add regression tests for the reasoning-only and function_call cases. Also drop the unused `processor` kwarg from the async GRPO call site. Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 4c2537b) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Add Args/Returns sections to image_to_data_url and encode_images_in_examples in nemo_rl/data/multimodal_utils.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit e9b89bc) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit d60004f) Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com>

Ignore text-only function call outputs during image extraction and seed the first trainable turn with images from the initial input messages. Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com>

Non-colocated 2n8g layout (vLLM TP=8 on node 1, Megatron TP=2/EP=8/CP=2 on node 2) for the single-turn Circle-Click NeMo-Gym environment. Model points at the HF repo (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16); train/eval data_path use /path/to/{train,eval}_dataset.jsonl placeholders so users wire in their own manifests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com>

The multimodal postprocessing path in _attach_multimodal_data_to_user_message assumes a placeholder-style processor: it reconstructs imgs_sizes / num_frames and builds the pixel_values PackedTensor with pad_to_max_shape=True. A non-placeholder VLM (e.g. Qwen2-VL / LLaVA-style) would silently produce wrong multimodal tensors instead of erroring. Fail loud at NemoGym.__init__ so the misconfiguration is caught at actor construction, well before any rollout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com>

…click recipe Adds the required defaults: ../../vlm_grpo_3B_megatron.yaml key so the recipe passes the configs-minimize-check pre-commit hook (all recipes under examples/configs/recipes/**/*.yaml must inherit from an exemplar). Drops the standalone header preamble; provenance now lives in the adjacent clevr/mmpr sibling recipes and the commit history. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com>

Signed-off-by: rohitrango <rohit.rango@gmail.com>

The circle-click recipe YAML landed without the driver script and suite entry that tests/unit/test_recipes_and_test_suites.py requires, so test_all_recipe_yamls_accounted_for_in_test_suites failed on a 229 vs 228 count mismatch. Add the missing driver. circle_click is a NeMo-Gym env, so it runs through run_grpo_nemo_gym.py rather than run_vlm_grpo.py, and the script regenerates its data via the resources server's generate_data.py (the committed example.jsonl has 5 rows and no agent_ref) with disjoint train/eval seeds. List it in disabled.txt rather than nightly.txt for now: the recipe has not been run end to end, so its reward threshold is an unvalidated smoke bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

(cherry picked from commit c3b8dacb3320214e8d5404fb4f31de01a63df2f8) Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

copy-pr-bot · 2026-08-06T18:30:49Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

rohitrango · 2026-08-07T19:10:59Z

Summary for the review above (the review body was lost on submit; GitHub does not allow adding one afterwards).

Reviewed with a team of agents (core data structures, GRPO/rollouts/async integration, config and guideline conformance, tests, an independent bug scan) plus adversarial verification passes that re-derived each finding from source before inclusion.

Scope. ~4,336 added lines (production +1,977 / tests +2,359). Roughly 1,030 of those belong to the payload-metrics subsystem: a 403-line module, a 327-line test file, and ~40 plumbing sites across 9 production files (115 references total). Its dependency direction is one-way — nothing in multimodal_utils.py, batched_data_dict.py, llm_message_utils.py, or nemo_gym.py imports it — and debug_payload_metrics is set true in zero configs repo-wide. Splitting it removes 2 of the 3 new user-facing config keys without affecting dedup behavior.

Correctness. One issue, at multimodal_utils.py:629: flattened_concat's dedup shortcut returns cls.concat(...), which produces sum(len(p)) rows, while the method's documented contract (and its doctest) is one row per input. It is sound only when every input has len == 1, and merge_segments:577-582 has a legacy passthrough that breaks that assumption for a mixed dedup/legacy batch. One-line fix; the suggested concat consolidation also resolves it.

bf16 pre-cast. Applied at 3 of 9 structurally identical get_multimodal_dict(as_tensors=False) call sites; grpo_sync.py:627 is the near-verbatim twin of a patched site and was not updated. The dtype is hardcoded rather than derived from policy.precision, and the workers do not re-cast. Validation for that commit is byte-count only, and the convergence campaigns predate it.

Unconditional changes. The vllm_content turn>0 handling at rollouts.py:871-872 and the vllm/utils.py:114-127 rework are not gated on deduplicate_multimodal_data. Tracing it: vllm_content is written only at processors.py:663/:681 at dataset-processing time, so before this change vLLM received the stale turn-0 prompt string at every turn > 0 and re-generated from turn 0. This is a fix, not a regression, but it currently reads as dedup plumbing and has no dedicated test.

Documentation. 29de7b52 deleted docs/design-docs/multimodal-deduplication.md (193 lines) and its docs/index.md entry. grep over docs/ returns zero hits for this feature, and the new YAML keys carry no comments.

Investigated and excluded after verification, listed so they are not re-raised: batch_size() strictness under allow_missing_packed_tensors; empty_rows_like metadata leaking into flag-off objects; the Gym per-turn image seeding path; the payload-restore clobber; and a suspected GenerationConfig regression that traced to already-merged #3401 rather than this PR.

Generated by Claude Code

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

Four follow-ups to the dedup review, none of which change production behavior for any shipped configuration. - test_grpo: add the required policy.precision to mock_grpo_components. bd4a2a2 replaced a hardcoded torch.bfloat16 with _policy_dtype(), which indexes policy_config["precision"] eagerly on every GRPO step, so the shared fixture failed 25 pre-existing tests with KeyError: 'precision'. - PackedTensor.to_dtype: skip non-floating-point segments. The cast allowlist was widened to pixel_values_videos without the float guard that was its stated precondition, leaving a string allowlist as the only thing keeping integer media metadata (grid sizes, frame counts) out of a float cast. - GenerationConfig: rename debug_payload_metrics to _debug_payload_metrics, matching the _pad_token_id / _mtp_weights_from_refit convention for keys populated internally rather than by the user. grpo.setup() overwrites the value unconditionally, so a user-set key was silently discarded. Readers now use bool(cfg.get(...)) with no invented default, matching vllm_worker.py's read of _mtp_weights_from_refit. - test_rollouts: the two multi-turn tests parametrized deduplicate_multimodal_data but asserted nothing that differed between legs. Capture the forwarded flag and the presence of the policy-ready media payload so both legs prove distinct behavior. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw

Follow-up on bd4a2a29: verified the responses to both reviews against the code. Nearly everything checks out — the deletions are clean (task-index recovery is equivalent across six resume scenarios, save_replay_buffer removal is behavior-identical for every in-tree config, and the flattened_concat guard turns the old silent row misalignment into a loud failure). We've resolved the threads that landed.

One new issue below: the bf16 → policy.precision fix, which was the right call, leaves test_grpo.py with 25 failing tests. Flagging it because CI hasn't run yet, so there's currently no signal.

Generated by Claude Code

…r attach helpers Follow-ups to the two open documentation/coverage threads. - PackedTensor: add a worked CSR example. The numbers are derived from a real G=3 expansion rather than transcribed, so six logical rows over nine segment references map onto three physical tensors. Describes the mechanism that actually ships (deep-copied rows plus _prepare_multimodal_sharing aliasing the media leaves, re-interned by concat), since PackedTensor.repeat_interleave was removed. - PackedTensor.to: record why a device move retains provenance while to_dtype mints fresh provenance -- concat re-interns on provenance alone, so a value-changing operation must not keep it. Notes that the mixed-device corollary is currently unreachable and would fail loudly. - test_rollouts: cover the two untested paths in attach_initial_nemo_gym_image_payloads and attach_image_model_inputs_to_message -- the "no user message" ValueError, that rollout token_ids survive while media is packed, and the no-op guards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw

approved up until my last 2 commits

yfw · 2026-08-11T05:42:13Z

/ok to test 90c467e

CI runs `pre-commit run --all-files`, which includes `ruff check --select I` (isort) and `ruff-format` at the pinned v0.9.9. Those are separate hooks from the `ruff check` governed by pyproject's `select = ["D", "F"]`, so this drift was not caught by a plain `ruff check`. Import ordering: multimodal_payload_metrics.py, test_rollouts.py, test_config_validation.py. Formatting: grpo.py, test_llm_message_utils.py. Mechanical only -- no logic changes. The one remaining format diff is inside the Automodel submodule, which is not tracked by this repo and so is out of scope for pre-commit here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw · 2026-08-11T06:19:06Z

/ok to test bd2a314

test_config_v2_same_as_v1 compares each v1 reference config against its Pydantic v2 model_dump and fails when the model introduces defaults the reference lacks. Adding deduplicate_multimodal_data and debug_payload_metrics to the GRPOConfig BaseModel therefore broke it: assert 2 == 0 + where 2 = len(['grpo.deduplicate_multimodal_data', 'grpo.debug_payload_metrics']) Per the config-conventions skill, a change to an exemplar covered by test_config_v2.py must also update the matching reference config. The trigger is adding fields to the BaseModel, not editing the exemplar YAML itself, so this applies even though the PR left examples/configs/grpo_math_1B.yaml untouched. Values and comments match the exemplar entries in vlm_grpo_3B.yaml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

yfw · 2026-08-11T13:04:57Z

/ok to test a23a8d1

addressed in follow up commits

rohitrango and others added 28 commits August 5, 2026 09:47

delete scratchspace configs

5b43d98

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 7ddf223) Signed-off-by: rohitrango <rohit.rango@gmail.com>

consolidate into one entrypoint

faa185a

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 7bf55af) Signed-off-by: rohitrango <rohit.rango@gmail.com>

reverted processor design to be consistent with vlm_grpo

98de6f8

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit b7fc746) Signed-off-by: rohitrango <rohit.rango@gmail.com>

chore: clean up multimodal processor plumbing

a50f772

Signed-off-by: rohitrango <rohit.rango@gmail.com>

change per-turn images to get results from tool-calls, etc (anything

f6300f8

except assistant) Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit 85a24b8) Signed-off-by: rohitrango <rohit.rango@gmail.com>

change non-default config option

a104ae6

Signed-off-by: rohitrango <rohit.rango@gmail.com> (cherry picked from commit d60004f) Signed-off-by: rohitrango <rohit.rango@gmail.com>

build: preserve main dependency configuration

d3e02b4

Signed-off-by: rohitrango <rohit.rango@gmail.com>

(chore): add copyright notice to test

428f5ab

Signed-off-by: rohitrango <rohit.rango@gmail.com>

(chore): undo vllm chat request change

33b70ec

Signed-off-by: rohitrango <rohit.rango@gmail.com>

fix: address NeMo Gym multimodal image indexing issues

ef3f163

Ignore text-only function call outputs during image extraction and seed the first trainable turn with images from the initial input messages. Signed-off-by: rohitrango <rohit.rango@gmail.com>

lint fixes

1c5bfbb

Signed-off-by: rohitrango <rohit.rango@gmail.com>

allow mixed (multimodal, text) batches from nemo-gym batch rollouts

062e999

Signed-off-by: rohitrango <rohit.rango@gmail.com>

chore: apply ruff format to llm_message_utils

0edbe98

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: rohitrango <rohit.rango@gmail.com>

minimized config

4bb7e20

Signed-off-by: rohitrango <rohit.rango@gmail.com>

Merge branch 'main' into rohit/gymv-mm-integration-v2

8ba3fe6

Merge branch 'main' into rohit/gymv-mm-integration-v2

f27dc6d

feat: deduplicate multimodal GRPO payloads

9195924

(cherry picked from commit c3b8dacb3320214e8d5404fb4f31de01a63df2f8) Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

chore: trim multimodal dedup scope

29de7b5

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

aroshanghias-nvd marked this pull request as ready for review August 6, 2026 18:36

aroshanghias-nvd changed the base branch from rohit/gymv-mm-integration-v2 to main August 7, 2026 10:57

aroshanghias-nvd mentioned this pull request Aug 7, 2026

feat(grpo): deduplicate multimodal payloads #3534

Closed

7 tasks

aroshanghias-nvd closed this Aug 7, 2026

aroshanghias-nvd reopened this Aug 7, 2026

chore: retrigger checks against main

1c31c47

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

rohitrango previously requested changes Aug 7, 2026

View reviewed changes

fix(multimodal): address dedup review feedback

bd4a2a2

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>

aroshanghias-nvd requested a review from a team as a code owner August 10, 2026 21:05

github-actions Bot added the Documentation Improvements or additions to documentation label Aug 10, 2026

yfw reviewed Aug 11, 2026

View reviewed changes

Comment thread nemo_rl/algorithms/grpo.py

yfw force-pushed the aroshanghias/multimodal-dedup-pr3414-v3 branch from 768b39f to 6bb0290 Compare August 11, 2026 05:40

yfw previously approved these changes Aug 11, 2026

View reviewed changes

Merge branch 'main' into aroshanghias/multimodal-dedup-pr3414-v3

90c467e

yfw added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Aug 11, 2026

yfw dismissed their stale review via bd2a314 August 11, 2026 06:18

yfw enabled auto-merge (squash) August 11, 2026 06:19

aroshanghias-nvd added the CI:L1 Run doctests, unit tests, and functional tests label Aug 11, 2026

yfw removed the CI:L1 Run doctests, unit tests, and functional tests label Aug 11, 2026

yfw disabled auto-merge August 11, 2026 16:03

yfw approved these changes Aug 11, 2026

View reviewed changes

yfw merged commit 96b3271 into NVIDIA-NeMo:main Aug 11, 2026
82 checks passed

Navigation Menu

Uh oh!

feat(grpo): deduplicate multimodal payloads - #3527

feat(grpo): deduplicate multimodal payloads#3527
yfw merged 37 commits into
NVIDIA-NeMo:mainfrom
aroshanghias-nvd:aroshanghias/multimodal-dedup-pr3414-v3

aroshanghias-nvd commented Aug 6, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohitrango commented Aug 7, 2026

Uh oh!

yfw left a comment

Uh oh!

Uh oh!

yfw left a comment

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aroshanghias-nvd commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deduplicate multimodal GRPO payloads

What this changes

Representation and safety contract

Quantified G=16 evidence

Nemotron Omni 30B, Megatron, async NeMo Gym

Qwen2.5-VL 3B, Megatron, native rollout

Qwen2.5-VL 3B, AutoModel, native rollout

Gemma 3 4B, AutoModel, transport-only diagnostic

Current-#3414 validation

Evaluation limits

Test plan

Uh oh!

copy-pr-bot Bot commented Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohitrango commented Aug 7, 2026

Uh oh!

yfw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yfw left a comment

Choose a reason for hiding this comment

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

yfw commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aroshanghias-nvd commented Aug 6, 2026 •

edited

Loading