[feat] support kimi_vl VLM model

[ShareLer]

Checklist Before Starting

• Search for similar PR(s).
  Some code will conflict with this PR [FEAT] Add the support for more VLMs(Gemma3) #1613

What does this PR do?

Add initial support for Kimi_vl;
Add sp patch for kimi_vl.

High-Level Design

Demonstrate the high-level design if this PR is complex.

Specific Changes

• Add some minor changes to be compatible with kimi_vl
• Add patch to support ulysses_sequence_parallel

API

Demonstrate how the API changes if any.

Usage Example

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$DATA_PATH/geo3k/test.parquet \
    data.val_files=$DATA_PATH/geo3k/test.parquet \
    data.train_batch_size=16 \
    data.max_prompt_length=2048 \
    data.max_response_length=4096 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    data.image_key=images \
    data.shuffle=False \
    +data.trust_remote_code=True \
    actor_rollout_ref.model.path=moonshotai/Kimi-VL-A3B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ulysses_sequence_parallel_size=2 \
    actor_rollout_ref.actor.ppo_mini_batch_size=8 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.01 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.model.enable_gradient_checkpointing=False \
    actor_rollout_ref.model.trust_remote_code=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=8\
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.enable_chunked_prefill=False \
    actor_rollout_ref.rollout.enforce_eager=False \
    actor_rollout_ref.rollout.free_cache_engine=False \
    actor_rollout_ref.rollout.n=8 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=1 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.val_before_train=False \
    trainer.critic_warmup=0 \
    trainer.logger=['console','wandb'] \
    trainer.project_name='Kimi_VL_test' \
    trainer.experiment_name='kimi_vl_grpo_geo3k_cp2' \
    trainer.n_gpus_per_node=8\
    trainer.nnodes=1\
    trainer.save_freq=50 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@

Test & Problem

During the dev, I discovered some issues, but they did not affect the code for this PR.
Existing problems：（with vllm==0.8.5.post1）

• Occasional errors of vllm

  File "/home/sharele/anaconda3/lib/python3.11/site-packages/vllm/v1/attention/backends/mla/common.py", line 504, in build
    self.page_size)
    ^^^^^^^^^^^^^^
AttributeError: 'MLACommonMetadataBuilder' object has no attribute 'page_size'

releated: vllm-project/vllm#16908
Reference this method to avoid the problem temporarily: vllm-project/vllm#16908 (comment)

• Garbled output from vllm under specific circumstances
  During test, I found that when SamplingParams.n > 1，vllm's output is some meaningless characters or keeps repeating. This will affect grpo.

releated: vllm-project/vllm#18378
Note: Using the Hopper architecture gpu can avoid this problem, but it is not clear whether there are still potential issues.

Training curve:
The training curve will comming soon after I solve the second problem.

Additional Info.

• Issue Number: support for kimi-vl？ #1428
• Training: FSDP
• Inference: vLLM

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks.
• Add [BREAKING] to the PR title if it breaks any API.
• Update the documentation about your changes in the docs.
• Add CI test(s) if necessary.

[hiyouga]

Overall looks good to me!

[ShareLer]

@hiyouga It seems that there are some problems with the CI environment. Could you help retrigger the failed CI?

[hiyouga]

No problem, let's merge