feat: Multimodal nemo gym compatible grpo pipeline - #3414

Merged

yfw merged 27 commits into

mainfrom

rohit/gymv-mm-integration-v2

Aug 7, 2026

Commits on Aug 5, 2026

feat(nemo-gym): support multimodal rollouts with tokenizer_config plumbing and chat_template parity
rohitrango
committed
b7fa6e3
feat(nemo-gym): plumb multimodal rollouts through the async single-controller pathway
rohitrango
committed
edb54a6
chore(examples): add Nemotron-Omni gym-v smoke configs (tangram + polygon-naming + multi-turn tool calling)
rohitrango
committed
4cb8023
fix(nemo-gym): drop unused nemo_gym_row arg from _postprocess_nemo_gym_to_nemo_rl_result
rohitrango
committed
37098a9
delete scratchspace configs
rohitrango
committed
5b43d98
consolidate into one entrypoint
rohitrango
committed
faa185a
reverted processor design to be consistent with vlm_grpo
rohitrango
committed
98de6f8
chore: clean up multimodal processor plumbing
rohitrango
committed
a50f772
change per-turn images to get results from tool-calls, etc (anything
rohitrango
committed
f6300f8
fix(nemo_gym): flush per-turn image bucket on any trainable item
rohitrango
committed
629a1d4
docs: add Google-style docstrings to image encoding helpers

rohitrango
and
claude
committed
171506f
change non-default config option
rohitrango
committed
a104ae6
build: preserve main dependency configuration
rohitrango
committed
d3e02b4
(chore): add copyright notice to test
rohitrango
committed
428f5ab
(chore): undo vllm chat request change
rohitrango
committed
33b70ec
fix: address NeMo Gym multimodal image indexing issues
rohitrango
committed
ef3f163
lint fixes
rohitrango
committed
1c5bfbb
allow mixed (multimodal, text) batches from nemo-gym batch rollouts
rohitrango
committed
062e999

Commits on Aug 6, 2026

chore: apply ruff format to llm_message_utils

rohitrango
and
claude
committed
0edbe98
feat(recipes): add Nemotron-Omni 30B circle-click 2n8g VLM-GRPO recipe

rohitrango
and
claude
committed
a70eacc
feat(nemo_gym): assert placeholder-style processor at actor init

rohitrango
and
claude
committed
68a538b
chore(recipes): inherit from vlm_grpo_3B_megatron exemplar in circle-click recipe

rohitrango
and
claude
committed
b9e1a27
minimized config
rohitrango
committed
4bb7e20
test(vlm): add circle-click gym driver script, disabled for now

yfw
and
claude
committed
aa5ea0b
Merge branch 'main' into rohit/gymv-mm-integration-v2
aroshanghias-nvd
authored
8ba3fe6
Merge branch 'main' into rohit/gymv-mm-integration-v2
rohitrango
authored
f27dc6d
Merge branch 'main' into rohit/gymv-mm-integration-v2
rohitrango
authored
234cdd7

Navigation Menu

Uh oh!

feat: Multimodal nemo gym compatible grpo pipeline - #3414

feat: Multimodal nemo gym compatible grpo pipeline#3414
yfw merged 27 commits into
mainfrom
rohit/gymv-mm-integration-v2

Commits on Aug 5, 2026

feat(nemo-gym): support multimodal rollouts with tokenizer_config plumbing and chat_template parity

feat(nemo-gym): plumb multimodal rollouts through the async single-controller pathway

chore(examples): add Nemotron-Omni gym-v smoke configs (tangram + polygon-naming + multi-turn tool calling)

fix(nemo-gym): drop unused nemo_gym_row arg from _postprocess_nemo_gym_to_nemo_rl_result

delete scratchspace configs

consolidate into one entrypoint

reverted processor design to be consistent with vlm_grpo

chore: clean up multimodal processor plumbing

change per-turn images to get results from tool-calls, etc (anything

fix(nemo_gym): flush per-turn image bucket on any trainable item

docs: add Google-style docstrings to image encoding helpers

change non-default config option

build: preserve main dependency configuration

(chore): add copyright notice to test

(chore): undo vllm chat request change

fix: address NeMo Gym multimodal image indexing issues

lint fixes

allow mixed (multimodal, text) batches from nemo-gym batch rollouts

Commits on Aug 6, 2026

chore: apply ruff format to llm_message_utils

feat(recipes): add Nemotron-Omni 30B circle-click 2n8g VLM-GRPO recipe

feat(nemo_gym): assert placeholder-style processor at actor init

chore(recipes): inherit from vlm_grpo_3B_megatron exemplar in circle-click recipe

minimized config

test(vlm): add circle-click gym driver script, disabled for now

Merge branch 'main' into rohit/gymv-mm-integration-v2

Merge branch 'main' into rohit/gymv-mm-integration-v2

Merge branch 'main' into rohit/gymv-mm-integration-v2