feat(vlm): route VLM GRPO through TQ trainer when data_plane.enabled - #2957

Merged

terrykong merged 33 commits into

mainfrom

zhiyul/tq_vlm

Sep 4, 2026

Commits on Sep 3, 2026

feat(vlm): route VLM GRPO through TQ trainer when data_plane.enabled

ZhiyuLi-Nvidia
and
claude
committed
ea7bdde
fix(grpo_sync): skip redundant iter-1 refit when weights already synced

ZhiyuLi-Nvidia
and
claude
committed
7715a48
fix(tq): thread VLM multimodal fields through TQ via torch.nested wire

ZhiyuLi-Nvidia
and
claude
committed
ef5b2cc
test(tq): full-VLM wire roundtrip + defensive-error coverage

ZhiyuLi-Nvidia
and
claude
committed
4969c77
test(tq): TQ-mediated VLM multimodal wire roundtrip

ZhiyuLi-Nvidia
and
claude
committed
653e1a3
fix(tq): ship multimodal fields to the train forward; guard dynbatch

ZhiyuLi-Nvidia
and
claude
committed
c28bfa9
fix(vlm): accept processor in grpo_train_sync for launcher parity

ZhiyuLi-Nvidia
and
claude
committed
75d2f17
fix(vlm): skip multimodal fields in the automodel seq-dim validator

ZhiyuLi-Nvidia
and
claude
committed
30118bd
fix(tq): address VLM data-plane review findings

ZhiyuLi-Nvidia
and
claude
committed
f7188dd
test(tq): cover the VLM wire boundary and dispatch parity

ZhiyuLi-Nvidia
and
claude
committed
0be1ad4
fix(tq): satisfy pyrefly on the multimodal wire encoder

ZhiyuLi-Nvidia
and
claude
committed
de3e0aa
refactor(tq): let TransferQueue own per-row shape; drop the wire companions

ZhiyuLi-Nvidia
and
claude
committed
c41c4c6
fix(tq): carry VLM multimodal row shapes without padding the wire

ZhiyuLi-Nvidia
and
claude
committed
8c1104b
docs(tq): note the ProcessorInterface refactor on the parity kwarg
ZhiyuLi-Nvidia
committed
e647333
docs(tq): correct from_wire's stale claim that it pads
ZhiyuLi-Nvidia
committed
25f78d9
docs(tq): correct the stale wire-form comment in get_multimodal_dict
ZhiyuLi-Nvidia
committed
0d1017f
refactor(tq): stop shipping a batch-wide pad target over the wire
ZhiyuLi-Nvidia
committed
e7ecafb
wip(tq): move the VLM omni nightly to megatron + mooncake
ZhiyuLi-Nvidia
committed
840f27b
fix(tq): address the second round of VLM data-plane review findings

ZhiyuLi-Nvidia
and
claude
committed
e12f4b3
test(tq): drop the TQ-only gate from the VLM omni mooncake nightly

ZhiyuLi-Nvidia
and
claude
committed
5e0fcfe
refactor(tq): reuse meta.subset in preshard; always reject wire form

ZhiyuLi-Nvidia
and
claude
committed
240de67
refactor(tq): dedupe launcher dispatch and tighten the VLM wire helpers

ZhiyuLi-Nvidia
and
claude
committed
e26f249
fix(tq): bind extract_field_schema's upstream before the closure

ZhiyuLi-Nvidia
and
claude
committed
b8b8fe4
test(vlm): add a 2-node Nemotron-Omni clevr Megatron recipe

ZhiyuLi-Nvidia
and
claude
committed
3895c1d
test(tq): point the mooncake VLM wrapper at the 2-node base recipe

ZhiyuLi-Nvidia
and
claude
committed
cc526f7
test: raise the nightly compute cap to 4266 GPU hours

ZhiyuLi-Nvidia
and
claude
committed
5827223

Commits on Sep 4, 2026

fix(tq): keep media-free packed fields in the leader-broadcast key set

ZhiyuLi-Nvidia
and
claude
committed
62ed912
fix(tq): guard the SC microbatch path and scope the multimodal union

ZhiyuLi-Nvidia
and
claude
committed
9a3dae9
docs(tq): correct the wire-form comments and the test inventory

ZhiyuLi-Nvidia
and
claude
committed
e52974e
test: fit the VLM TQ nightlies under the existing compute cap

ZhiyuLi-Nvidia
and
claude
committed
fb7800c
test(tq): give both VLM TQ wrappers their own logprob gate

ZhiyuLi-Nvidia
and
claude
committed
6ed7a25
fix(test): follow the launcher dispatch and the worker guard

ZhiyuLi-Nvidia
and
claude
committed
2f407aa
chore: untrack recipe run outputs and stray agent config

ZhiyuLi-Nvidia
and
claude
committed
e2842d3

Navigation Menu

Uh oh!

feat(vlm): route VLM GRPO through TQ trainer when data_plane.enabled - #2957

feat(vlm): route VLM GRPO through TQ trainer when data_plane.enabled#2957
terrykong merged 33 commits into
mainfrom
zhiyul/tq_vlm

Commits on Sep 3, 2026

feat(vlm): route VLM GRPO through TQ trainer when data_plane.enabled

fix(grpo_sync): skip redundant iter-1 refit when weights already synced

fix(tq): thread VLM multimodal fields through TQ via torch.nested wire

test(tq): full-VLM wire roundtrip + defensive-error coverage

test(tq): TQ-mediated VLM multimodal wire roundtrip

fix(tq): ship multimodal fields to the train forward; guard dynbatch

fix(vlm): accept processor in grpo_train_sync for launcher parity

fix(vlm): skip multimodal fields in the automodel seq-dim validator

fix(tq): address VLM data-plane review findings

test(tq): cover the VLM wire boundary and dispatch parity

fix(tq): satisfy pyrefly on the multimodal wire encoder

refactor(tq): let TransferQueue own per-row shape; drop the wire companions

fix(tq): carry VLM multimodal row shapes without padding the wire

docs(tq): note the ProcessorInterface refactor on the parity kwarg

docs(tq): correct from_wire's stale claim that it pads

docs(tq): correct the stale wire-form comment in get_multimodal_dict

refactor(tq): stop shipping a batch-wide pad target over the wire

wip(tq): move the VLM omni nightly to megatron + mooncake

fix(tq): address the second round of VLM data-plane review findings

test(tq): drop the TQ-only gate from the VLM omni mooncake nightly

refactor(tq): reuse meta.subset in preshard; always reject wire form

refactor(tq): dedupe launcher dispatch and tighten the VLM wire helpers

fix(tq): bind extract_field_schema's upstream before the closure

test(vlm): add a 2-node Nemotron-Omni clevr Megatron recipe

test(tq): point the mooncake VLM wrapper at the 2-node base recipe

test: raise the nightly compute cap to 4266 GPU hours

Commits on Sep 4, 2026

fix(tq): keep media-free packed fields in the leader-broadcast key set

fix(tq): guard the SC microbatch path and scope the multimodal union

docs(tq): correct the wire-form comments and the test inventory

test: fit the VLM TQ nightlies under the existing compute cap

test(tq): give both VLM TQ wrappers their own logprob gate

fix(test): follow the launcher dispatch and the worker guard

chore: untrack recipe run outputs and stray agent config