feat(grpo): deduplicate multimodal payloads - #3533

Closed

aroshanghias-nvd wants to merge 30 commits into

NVIDIA-NeMo:mainfrom

aroshanghias-nvd:aroshanghias/multimodal-dedup-pr3414-v3

Commits on Aug 5, 2026

feat(nemo-gym): support multimodal rollouts with tokenizer_config plumbing and chat_template parity
rohitrango
committed
b7fa6e3
feat(nemo-gym): plumb multimodal rollouts through the async single-controller pathway
rohitrango
committed
edb54a6
chore(examples): add Nemotron-Omni gym-v smoke configs (tangram + polygon-naming + multi-turn tool calling)
rohitrango
committed
4cb8023
fix(nemo-gym): drop unused nemo_gym_row arg from _postprocess_nemo_gym_to_nemo_rl_result
rohitrango
committed
37098a9
delete scratchspace configs
rohitrango
committed
5b43d98
consolidate into one entrypoint
rohitrango
committed
faa185a
reverted processor design to be consistent with vlm_grpo
rohitrango
committed
98de6f8
chore: clean up multimodal processor plumbing
rohitrango
committed
a50f772
change per-turn images to get results from tool-calls, etc (anything
rohitrango
committed
f6300f8
fix(nemo_gym): flush per-turn image bucket on any trainable item
rohitrango
committed
629a1d4
docs: add Google-style docstrings to image encoding helpers

rohitrango
and
claude
committed
171506f
change non-default config option
rohitrango
committed
a104ae6
build: preserve main dependency configuration
rohitrango
committed
d3e02b4
(chore): add copyright notice to test
rohitrango
committed
428f5ab
(chore): undo vllm chat request change
rohitrango
committed
33b70ec
fix: address NeMo Gym multimodal image indexing issues
rohitrango
committed
ef3f163
lint fixes
rohitrango
committed
1c5bfbb
allow mixed (multimodal, text) batches from nemo-gym batch rollouts
rohitrango
committed
062e999

Commits on Aug 6, 2026

chore: apply ruff format to llm_message_utils

rohitrango
and
claude
committed
0edbe98
feat(recipes): add Nemotron-Omni 30B circle-click 2n8g VLM-GRPO recipe

rohitrango
and
claude
committed
a70eacc
feat(nemo_gym): assert placeholder-style processor at actor init

rohitrango
and
claude
committed
68a538b
chore(recipes): inherit from vlm_grpo_3B_megatron exemplar in circle-click recipe

rohitrango
and
claude
committed
b9e1a27
minimized config
rohitrango
committed
4bb7e20
test(vlm): add circle-click gym driver script, disabled for now

yfw
and
claude
committed
aa5ea0b
Merge branch 'main' into rohit/gymv-mm-integration-v2
aroshanghias-nvd
authored
8ba3fe6
Merge branch 'main' into rohit/gymv-mm-integration-v2
rohitrango
authored
f27dc6d
feat: deduplicate multimodal GRPO payloads
aroshanghias-nvd
committed
9195924
chore: trim multimodal dedup scope
aroshanghias-nvd
committed
29de7b5
perf(grpo): pre-cast pixel payloads to bf16
aroshanghias-nvd
committed
d95bb1a

Commits on Aug 7, 2026

Merge main into multimodal dedup branch
aroshanghias-nvd
committed
36aa640