[WIP][feat] Initial support for VLMs, add Qwen2VL SFT test and Qwen2.5VL GRPO test

[antoinegg1]

What does this PR do？

This PR migrates the feature of SFT and GRPO on VLMs in our implementation in AreaLite. We have validated this feature using Qwen2-VL model and Qwen2.5-VL model on 8*H100 GPUs. The configuration and data processing script are provided along this PR for easy reproducing.

How to reproduce?

Download: Qwen2-VL-7B, Qwen2.5-VL-7B BUAADreamer/clevr_count_70k (directly use name is also supported)

# for SFT
torchrun --nnodes 1 --nproc-per-node 8 examples/arealite/clevr_count_70k_sft.py --config examples/arealite/configs/clevr_count_70k_sft.yaml 
# for GRPO
WANDB_BASE_URL=f"{your_wandb_base_url}" WANDB_API_KEY=f"{your_wandb_api_key}" python3 -m arealite.launcher.local examples/arealite/clevr_count_70k_grpo.py --config examples/arealite/configs/clevr_count_70k_grpo.yaml

Dependencies

transformers>=4.53.1

Major Changes

dataflow and processing

在dataset/init.py设计了统一的数据集处理函数，在config中添加了path参数

def get_custom_dataset(
    path: str,
    rank: int,
    world_size: int,
    training_type: str= "sft",
    split: Optional[str] = None,
    tokenizer: Optional[transformers.PreTrainedTokenizerFast] = None,
    processor: Optional[transformers.AutoProcessor] = None,
    ):

在api/io_struct.py 添加了vlm数据结构
在workflow/vision_rlvr.py中添加了vision数据的process

New workflow for multimodal grpo

在base_hf_engine, fsdp_engine中添加了处理vision的分支逻辑

Other Changes

在util.py 中添加和修改了一些函数以适应vision数据流
在arealite/tests中添加了一些vision的测试脚本
在example/arealite中添加了测试样例

Perfomance

使用Qwen2.5-VL-3B-Instruct model，在clevr_count 70k 数据集抽样30%的subset上，GRPO训练一个epoch，耗时约5小时，最后accuracy达到0.96。

[garrett4wade]

should revert to the version in the "lite" branch