[Contribute] Migrate LeRobot VLA Models to XLeRobot

[Vector-Wangel]

📋 Issue Description

Migrate existing VLA models from LeRobot (originally for single SO-101 arms) to work with XLeRobot dual-arm mobile platform.

🎯 What We Need

Target Models Options: DP3, ACT, Pi0, SmolVLA, GR00T

Requirements:

• Model inference on XLeRobot hardware/simulation
• Action space mapping from single arm to dual-arm + base
• Working demo with pick-and-place task
• Basic documentation

🔧 Expected Approach

1. Study LeRobot VLA implementations (refer to [LeRobot official website](https://github.com/huggingface/lerobot))
2. Create action space adapter for XLeRobot format
3. Test with pre-trained checkpoints
4. Create demo script showing task execution

📝 Acceptance Criteria

• Successfully run at least 1 VLA model on XLeRobot
• Demo video
• Basic setup documentation
• Example usage script

💡 Resources

• [LeRobot Models](https://github.com/huggingface/lerobot)
• XLeRobot Examples
• Dual-arm Control Reference

🤝 Propose Your Approach

Comment with your implementation plan! Include:

• Which models you'll start with
• How you'll handle action space mapping
• Timeline estimate

---

This will bring powerful VLA capabilities to our dual-arm mobile platform! 🤖

[hyy02]

Implementation plan:

• Models start with: ACT in Maniskill, SmolVLA in real world
• Timeline estimate:
  8.29: Begin to implement it.
  9.2: Finish the first task for the benchmark(ReplicaCAD with pick and place task, Guidence documentation for other developers)
  9.7: The simulation result for Maniskill, (if result is good, will finished the example usage script)
  9.12: SmolVLA real world experiment, (if result is good, will finished the example usage script)

I will show the experiment result and my process under this issue, in this way we can verify the timeline, other models may come after these two.

[hyy02]

Hello, I roughly see the documentation just now, I think that I need to change the time line. Since XLerobot doesn't have a simulation benchmark now, so I think I need to build it first, then we can test the model. I roughly see the code just now, I'd like to build begin with the scene with ReplicaCAD first and do some simple task like place and pick, I think the main part for it is the motion planning, a naive idea I have is control the robot base first, then plan the robotics arms. Do you have better ideas? Besides, since the work become more than I expected, I need to change the time table, sorry for that 😂.

And also, there is some recommendation for the simulation part, I will help with improve the documentation after finishing build the benchmark :

1. I see that we use fetch for alternative, but I think that we can use our robot name xlerobot, which is compitable with the maniskill.
2. The installation part recommend users to change the code in conda, I think maybe we can install it in a developer mode like pip install -e ., or just fork a maniskill repo for xlerbot so that we don't need to change the conda environment code.

I'll begin to work on it, and I'll share my process and problems meeting with you, thanks for your help!

[hyy02]

Oh, I see we have already change the fetch to xlerbot, nice update, I need to pull the repo 😂 😂 😂

[Vector-Wangel]

Hi, thanks for the update!

For the simulation benchmark, I am planning to integrate XLeRobot firstly with Maniskill HAB. It should be sufficient and convenient for us to start working on some basic moblie manipulation. No worries for the time since I also sort of pause my development on XLeRobot to focus on my own research for a while.

I have been thinking about pip install -e., for both maniksill and lerobot. But since they are also actively updating, my concern is that we will also have to update our own repo as long as they updates. I think for now I will quickly make a pull request to maniskill to list xlerobot as one of the officially supported robots, so that we can just install maniskill directly to play with XLeRobot. Later I will also pull request to lerobot to have official support.

Right now my opinion is that we don't need a specific python library to maintain XLeRobot but to integrate it to larger platforms like Maniskill, lerobot, Isaac Lab, Mujoco, etc.

I am currently working on my research project "OpenCage", which will be an opensource repo with pip install -e., that formulate a new framework for state-representation and training, with example support for XLeRobot.

[hyy02]

Thanks for your reply.

1. Benchmark building: In ManiSkill, our typical workflow is to inherit the Env environment, customize the scene_builder, modify the agent, construct motion planning, implement the solving strategy, debug the success criteria, and perform data collection. ManiSkill-Hab provides a reference example that can help us complete this process more efficiently. I'm wondering if you begin the benchmark building with Maniskill Hab right now, is it possible for me to help you with building it?
2. For the ideat to intergrate XLerobot to maniskill official repo, I thinks it's make sense. I'll take this into consideration while developing.
3. For the project OpenCage, it's nice, looking forward to seeing it.

[Vector-Wangel]

I may not begin implementing the Maniskill HAB at least for these two weeks. You can definitely get started first, thanks!

[hyy02]

Hi, I'm training the pick policy for XLerobot in ManiSkill-Hab, but the reward in wandb doesn’t look good. I’d like to verify the qpos mapping for XLerobot. For example, with the XLerobot controller pd_joint_delta_pos_dual_arm, I noticed it uses the DictControllerConfig in ManiSkill. In the simulation’s keyboard control code, I found the joint mapping function, so I believe the mapping is as follows:

class XLeRobotActionWrapper(gym.ActionWrapper):
    def __init__(
        self, 
        env, 
        stationary_base=False, 
        stationary_arm1=False, 
        stationary_arm2=False,
        stationary_head=True
    ):
      
        self._stationary_base = stationary_base
        self._stationary_arm1 = stationary_arm1
        self._stationary_arm2 = stationary_arm2
        self._stationary_head = stationary_head
        super().__init__(env)

    def action(self, action):
        if self._stationary_base:
            action[..., 0:2] = 0
            
        if self._stationary_arm1:
            # Fix first arm's 5 DOF
            action[..., [3, 6, 9, 11, 13]] = 0
            
        if self._stationary_arm2:
            # Fix second arm's 5 DOF
            action[..., [4, 7, 10, 12, 14]] = 0

        if self._stationary_head:
            action[..., 14:16] = 0
            
        return action

is my understanding right?

[Vector-Wangel]

Hi, I'm training the pick policy for XLerobot in ManiSkill-Hab, but the reward in wandb doesn’t look good. I’d like to verify the qpos mapping for XLerobot. For example, with the XLerobot controller pd_joint_delta_pos_dual_arm, I noticed it uses the DictControllerConfig in ManiSkill. In the simulation’s keyboard control code, I found the joint mapping function, so I believe the mapping is as follows:

class XLeRobotActionWrapper(gym.ActionWrapper):
def init(
self,
env,
stationary_base=False,
stationary_arm1=False,
stationary_arm2=False,
stationary_head=True
):

    self._stationary_base = stationary_base
    self._stationary_arm1 = stationary_arm1
    self._stationary_arm2 = stationary_arm2
    self._stationary_head = stationary_head
    super().__init__(env)

def action(self, action):
    if self._stationary_base:
        action[..., 0:2] = 0
        
    if self._stationary_arm1:
        # Fix first arm's 5 DOF
        action[..., [3, 6, 9, 11, 13]] = 0
        
    if self._stationary_arm2:
        # Fix second arm's 5 DOF
        action[..., [4, 7, 10, 12, 14]] = 0

    if self._stationary_head:
        action[..., 14:16] = 0
        
    return action

is my understanding right?

Not fully correct. I think the grippers and head is wrong?

Here's the original mapping function from maniskill teleop demo example:

def get_mapped_joints(robot):
    """
    Get the current joint positions from the robot and map them correctly to the target joints.
    
    The mapping is:
    - full_joints[0,2] → current_joints[0,1] (base x position and base rotation)
    - full_joints[3,6,9,11,13] → current_joints[2,3,4,5,6] (first arm joints)
    - full_joints[4,7,10,12,14] → current_joints[7,8,9,10,11] (second arm joints)
    
    Returns:
        np.ndarray: Mapped joint positions with shape matching the target_joints
    """
    if robot is None:
        return np.zeros(16)  # Default size for action
        
    # Get full joint positions
    full_joints = robot.get_qpos()
    
    # Convert tensor to numpy array if needed
    if hasattr(full_joints, 'numpy'):
        full_joints = full_joints.numpy()
    
    # Handle case where it's a 2D tensor/array
    if full_joints.ndim > 1:
        full_joints = full_joints.squeeze()
    
    # Create the mapped joints array with correct size
    mapped_joints = np.zeros(16)
    
    # Map the joints according to the specified mapping
    if len(full_joints) >= 15:
        # Base joints: [0,2] → [0,1]
        mapped_joints[0] = full_joints[0]  # Base X position
        mapped_joints[1] = full_joints[2]  # Base rotation
        
        # First arm: [3,6,9,11,13] → [2,3,4,5,6]
        mapped_joints[2] = full_joints[3]
        mapped_joints[3] = full_joints[6]
        mapped_joints[4] = full_joints[9]
        mapped_joints[5] = full_joints[11]
        mapped_joints[6] = full_joints[13]
        
        # Second arm: [4,7,10,12,14] → [7,8,9,10,11]
        mapped_joints[7] = full_joints[4]
        mapped_joints[8] = full_joints[7]
        mapped_joints[9] = full_joints[10]
        mapped_joints[10] = full_joints[12]
        mapped_joints[11] = full_joints[14]

        # Grippers
        mapped_joints[12] = full_joints[15]
        mapped_joints[13] = full_joints[16]

        # Head
        mapped_joints[14] = full_joints[5]
        mapped_joints[15] = full_joints[8]
    
    return mapped_joints

[hyy02]

Oh, sorry — I realized I made a mistake. I didn’t look at the code carefully enough, that was my fault. Thanks a lot for pointing it out. I’m curious though: how did you figure out the correct mapping? Since the joints aren’t continuous, did you just debug it directly in the simulation?

Right now, I’m following the instructions in this issue
, and I also asked the author about training the pick policy, which may require around 50M samples (discussion here
). I’m currently training it on a single 4090D GPU. Since the mshab branch doesn’t seem to have clear documentation for multi-GPU training (related PR
), it might take nearly a week for training. I plan to review the code again, retrain today, and also try to implement multi-GPU training myself.

Do you have any advice? Thanks a lot!