[FSDP, VLM] feat: add vlm training for FSDP

[nanjiangwill]

Goal: Support VLM training on slime with FSDP

TODO

• model support
• • add support for class AutoModelForVision2Seq/AutoModelForImageTextToText
• • support Qwen2.5-VL/Qwen3-VL(we just use default hf) new-feature TBD @WindowsXp-Beta
• support dataset - refer to examples in examples
• • for single turn -> geo3k, how verl did it [sglang] feat: add multimodal input to multiturn async rollout verl-project/verl#2014 @coding-famer @nanjiangwill
• • for multi turn -> deepeyes, how verl did it [recipe] feat: add deepeyes recipe verl-project/verl#2398
• single turn VLM training
• • support image in prompt/data, image processing, data pipeline @coding-famer @nanjiangwill
• • dataset properly handle multimodal info @coding-famer
• • rollout properly handle multimodal info @coding-famer @nanjiangwill
• • training(fsdp) properly handle multimodal info @nanjiangwill this
• • able to train on geo3k
• multi turn VLM training @minleminzui
• • data pipeline for new images generated during the rollout stage

[zhuzilin]

we shouldn't try to pass the pixel values from sglang to megatron, but instead maybe re-process the image from the training side is better.

[coding-famer]

Hi @nanjiangwill, glad to see this important feature! I'd love to help with this PR if you're open to collaborating. I can help with supporting qwen-vl models and geo3k training example. Please LMK if you are happy with collaborating!

[nanjiangwill]

Hi @nanjiangwill, glad to see this important feature! I'd love to help with this PR if you're open to collaborating. I can help with supporting qwen-vl models and geo3k training example. Please LMK if you are happy with collaborating!

hi @coding-famer, that will be amazing! can i have ur email/contact number so i can reach out to you?

[coding-famer]

Hi @nanjiangwill, glad to see this important feature! I'd love to help with this PR if you're open to collaborating. I can help with supporting qwen-vl models and geo3k training example. Please LMK if you are happy with collaborating!

hi @coding-famer, that will be amazing! can i have ur email/contact number so i can reach out to you?

Have sent you an email!

[dongyuanjushi]

Hi @nanjiangwill and @coding-famer, I'd love to help with this PR about the multi-turn part if you are open to this.

[nanjiangwill]

Hi @nanjiangwill and @coding-famer, I'd love to help with this PR about the multi-turn part if you are open to this.

heyy thanks for reaching out! can i have your email?

[dongyuanjushi]

Hi @nanjiangwill and @coding-famer, I'd love to help with this PR about the multi-turn part if you are open to this.

heyy thanks for reaching out! can i have your email?

Have sent the email!

[zhaochenyang20]

我的神 @nanjiangwill 😭

[zhaochenyang20]

牛逼！

[aJupyter]

请问大概什么时候才会合并呀

[zhaochenyang20]

请问大概什么时候才会合并呀

肯定会早于星际之门集群建设好

[jhinpan]

Could the code for multimodal processing be merged as soon as possible? The support for multi-turn is a separate module and can be merged later. Otherwise, this will affect the issues based on the multimodal model and merge code.

This PR will be merge in the next few days. Pls just give us some time to do final check and review.

[zhaochenyang20]

🈚️敌！

[jhinpan]

We ran three experiments comparing different reward model configurations on 8*H100:

1. geo3k RM with tol=0.05 (original)
2. geo3k RM with tol=0.0 (strict matching)
3. default math RM

Results showed all three configurations perform similarly, indicating:

• The tolerance parameter has minimal impact on training outcomes
• The geo3k-specific reward model provides no advantage over the default math RM

I will later remove them. Just leave a note here to mark:

[zhuzilin]

LGTM! Left some minor comments.

[zhuzilin]

here we need sth like:

packed_batch = {
    "tokens": ...,
}
if multimodal_inputs:
    for key, mm_tensor in multimodal_inputs[i].items():
        ...
    packed_batch.extend({
        "multimodal_inputs": multimodal_data,
        "multimodal_num_items": multimodal_num_items,
    })
result.append(packed_batch)

[jhinpan]

used .update() instead of .extend() since dictionaries use update to merge key-value pairs.

[zhuzilin]

hmm.. it seems better to move this type conversion into the custom reward model? otherwise, it may influence the other users who are using dense rm for rlhf.

[jhinpan]

move float16 specifically to notes in geo3k_vlm example

[simondong1]

Hi @nanjiangwill, this is awesome! I am also working on RL with VLMs. Would love to contribute and collaborate on further works! my email is simondong0919 at gmail.com love to get in touch

[adol001]

@nanjiangwill @zhuzilin

It looks like this commit changed the behavior of the apply_chat_template parameter. As a result, this parameter becomes useless during dataset construction: instead of applying the chat template, the data from prompt_key is fed directly into something like:

messages = [{"role": "user", "content": messages}]

In custom generate scenarios, do you realize what this can lead to?

You’re effectively replacing what used to be a str with a dict (or even a list of dicts). If downstream code doesn’t explicitly validate types, it will just get stuffed into the prompt and you end up with outputs like:

First rollout sample: ['<|im_start|>user\n[{'role': 'user', 'content':

This kind of change alters the default behavior in a non-backward-compatible way. At the very least, compatibility with existing users should be considered — and there should be warnings or clear notices. In my case, it indirectly caused training quality to degrade. I wouldn’t have noticed this logic change if it weren’t for a new training job that made the issue obvious.

[zhuzilin]

@adol001 really sorry about this, we will revert this change.

[nanjiangwill]

@adol001 sorry again about this, we have fixed the issue with #1232 and #1234