[recipe] feat: add Experimental VLA RL Support

[The-Hierophant]

What does this PR do?

Experimental VLA RL Support

This recipe introduces experimental support for training SimpleVLA-OFT, a VLA model.

A key challenge in VLA RL training, which differs from standard LLM RL training, is that the environment/simulation phase has a higher computational overhead than the generation phase. To achieve high efficiency, RL in this context requires an effective environment scheduling mechanism in addition to verl's existing efficient training and inference scheduling. The goal is to reduce the inefficiency caused by the environment and the model's generation process waiting on each other.

The core computational model of this PR is inspired by the pipeline parallelism design from RLinf. It aims to overlap the environment's execution time with the model's generation time, thereby maximizing environment utilization.

This PR also proposes a future direction: creating a unified Env class. This class would encapsulate functionalities like tool calling, MCP, etc., under a single interface. The environment would manage its state internally, allowing the agent to communicate simply by calling step(action) to submit an action and receive an observation.

Currently, this code is located independently within the recipes folder. Much of the design is tightly coupled with the SimpleVLA model and the Libero environment, serving as an initial version for demonstration and discussion.

Supported Simulators

Simulator	Env Name	Difference	Benchmark data source
Mujoco	LiberoEnv	1. init task from init_states in Libero dataset 2. each env can have different tasks	https://github.com/Lifelong-Robot-Learning/LIBERO
IsaacSim	IsaacEnv	1. init task from random states, which has more variety than init_states in dataset 2. each sim process must using the same task for its envs	https://huggingface.co/datasets/china-sae-robotics/IsaacLabPlayGround_Dataset

Hardware Requirements

• Simulator GPU: NVIDIA L20 or L40 with 48GB memory and RT Cores

Notes:

1. Mujoco can failback to CPU mode with degraded performance if no RT Cores is available
2. IsaacSim only support GPU with RT Cores
3. RTX GPU will be supported in the future release with remote deployment feature, but it can not work with colocated mode because of the limitation of GPU memory capacity.

Docker image

The Isaac Lab support for libero dataset depends on RobotLearningLab project from The Isaac Lab Project Developers team. The project is in the process of being public available and is currently build in this image with BSD-3-Clause license.

recipe/vla/run_simpleVLA_libero_grpo.sh is the example of training SimpleVLA-OFT with this image:

vemlp-cn-shanghai.cr.volces.com/preset-images/verl_vla:preview_vla_0.1

Disaggregation Mode for Train-Rollout / Simulation

Disaggregate Train-Rollout workers and Simulation workers into different nodes.

To enable disaggregation mode for Train-Rollout nodes and Simulation nodes, we need to establish ray connection before running verl.

• On Train-Rollout node (default main node):

ray start --head --dashboard-host=0.0.0.0 --resources='{"train_rollout": 1}'

• On Simulation node:

ray start --address='<main_node_ip>:6379' --resources='{"sim": 1}'

Then run verl on main node only. See run_simpleVLA_isaac_disagg.sh for example.

• env.disagg_sim.enable=True enable disagg mode
• trainer.n_env_gpus_per_node GPUs for simulaton per node
• trainer.n_rollout_gpus_per_node GPUs for train-rollout node
• env.disagg_sim.nnodes sim node num
• trainer.nnodes train-rollout node num

Tips: you can run the following command on the sim node to check whether sim workers are scheduled up

python -c "import ray; ray.init(address=\"<main_node_ip>:6379\"); print(ray._private.state.available_resources_per_node())"

If you see output pattern like "'train_rollout': 0.9992" and "'sim': 0.9992", the sim workers are scheduled up successfully
The actual value depends on your GPUs per node, usually <1 - 1e-4 * num_gpus>

References:

• https://github.com/PRIME-RL/SimpleVLA-RL
• https://github.com/RLinf/RLinf

Checklist Before Starting

• Search for similar PRs. Paste at least one query link here: ...
• Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
  • {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
  • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
  • {type} is in feat, fix, refactor, chore, test
  • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
  • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

Using libero dataset with openvla-oft model in batch 8, the result is the same with data from SimpleVLA-RL paper (batch 64):

### API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• [ x] Read the Contribute Guide.
• [ x] Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• [ x] Add / Update the documentation.
• [ x] Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• [ x] Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[wuxibin89]

@The-Hierophant @chenhaiq Is it possible to move verl/models/openvla_oft and env to recipe/vla folder? VLA is a emerging field and is quit different from LLM/VLM model, ideally we should separate it to a new project, e.g verl-vla.

BTW, we're planning to move recipe to a new project: verl-recipe.