Skip to content

Commit 994979c

Browse files
njawale42kellyguo11pascal-roth
authored
Adds SkillGen framework to Isaac Lab with cuRobo support (#3303)
## Description This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility. **Note:** - Please look at the cuRobo usage license ![here](docs/licenses/dependencies/cuRobo-license.txt) - Please look at updated isaacsim license ![here](docs/licenses/dependencies/isaacsim-license.txt) ### Technical Implementation: **Annotation Framework:** - Manual subtask start annotations to cleanly separate skill execution from motion-planning segments - Consistent trajectory segmentation for downstream dataset consumers **Motion Planning:** - **Base Motion Planner (Extensible):** - Introduces a reusable planner interface for uniform integration: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py` - Defines a minimal, consistent API for planners: - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc. - The cuRobo planner inherits from this base class. - New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline. - **CuRobo Planner** (GPU-accelerated, collision-aware): - Multi-phase planning: approach → contact → retreat - Dynamic object attach/detach and contact-aware sphere management - Real-time world synchronization between Isaac Lab and cuRobo - Configurable collision filtering for contact phases - **Tests**: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py` - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py` - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py` **Data Generation Pipeline:** - Automated dataset generation with precise skill-based segmentation - Integrates with existing observation/action spaces - Supports multi-env parallel collection with cuRobo-backed planning **Visualization and Debugging:** - Rerun-based 3D visualization for trajectory/collision inspection - Real-time sphere visualization for collision boundaries and contact phases ### Dependencies: - **cuRobo**: motion planning and collision checking - **Rerun**: 3D visualization and debugging ### Integration: This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users. ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] This change requires a documentation update ## Screenshot ### SkillGen Data Generation <table> <tr> <td align="center"><strong>Cube Stacking SkillGen Data Generation</strong></td> <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae" alt="Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e" alt="Bin Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ### Bin Cube Stacking Behavior Cloned Policy ![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9) ### Rerun Integration ![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0) ### Motion Planner Tests <table> <tr> <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td> <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db" alt="Obstacle Avoidance cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde" alt="Cube Stack End-to-End cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]>
1 parent bc7e5d7 commit 994979c

36 files changed

+6061
-175
lines changed

.github/actions/docker-build/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ inputs:
1818
required: true
1919
dockerfile-path:
2020
description: 'Path to Dockerfile'
21-
default: 'docker/Dockerfile.base'
21+
default: 'docker/Dockerfile.curobo'
2222
required: false
2323
context-path:
2424
description: 'Build context path'

CONTRIBUTORS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ Guidelines for modifications:
101101
* Miguel Alonso Jr
102102
* Mingyu Lee
103103
* Muhong Guo
104+
* Neel Anand Jawale
104105
* Nicola Loi
105106
* Norbert Cygiert
106107
* Nuoyan Chen (Alvin)

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,8 @@ dependencies and assets are present in the [`docs/licenses`](docs/licenses) dire
202202
203203
Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](docs/licenses/dependencies/isaacsim-license.txt) for information on Isaac Sim licensing.
204204
205+
Note that the `isaaclab_mimic` extension requires cuRobo, which has proprietary licensing terms that can be found in [`docs/licenses/dependencies/cuRobo-license.txt`](docs/licenses/dependencies/cuRobo-license.txt).
206+
205207
## Acknowledgement
206208
207209
Isaac Lab development initiated from the [Orbit](https://isaac-orbit.github.io/) framework. We would appreciate if

docker/Dockerfile.curobo

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
2+
# All rights reserved.
3+
#
4+
# SPDX-License-Identifier: BSD-3-Clause
5+
6+
# Nvidia Dockerfiles: https://github.com/NVIDIA-Omniverse/IsaacSim-dockerfiles
7+
# Please check above link for license information.
8+
9+
# Base image
10+
ARG ISAACSIM_BASE_IMAGE_ARG
11+
ARG ISAACSIM_VERSION_ARG
12+
FROM ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} AS base
13+
ENV ISAACSIM_VERSION=${ISAACSIM_VERSION_ARG}
14+
15+
# Set default RUN shell to bash
16+
SHELL ["/bin/bash", "-c"]
17+
18+
# Adds labels to the Dockerfile
19+
LABEL version="2.1.1"
20+
LABEL description="Dockerfile for building and running the Isaac Lab framework inside Isaac Sim container image."
21+
22+
# Arguments
23+
# Path to Isaac Sim root folder
24+
ARG ISAACSIM_ROOT_PATH_ARG
25+
ENV ISAACSIM_ROOT_PATH=${ISAACSIM_ROOT_PATH_ARG}
26+
# Path to the Isaac Lab directory
27+
ARG ISAACLAB_PATH_ARG
28+
ENV ISAACLAB_PATH=${ISAACLAB_PATH_ARG}
29+
# Home dir of docker user, typically '/root'
30+
ARG DOCKER_USER_HOME_ARG
31+
ENV DOCKER_USER_HOME=${DOCKER_USER_HOME_ARG}
32+
33+
# Set environment variables
34+
ENV LANG=C.UTF-8
35+
ENV DEBIAN_FRONTEND=noninteractive
36+
37+
USER root
38+
39+
# Install dependencies and remove cache
40+
RUN --mount=type=cache,target=/var/cache/apt \
41+
apt-get update && apt-get install -y --no-install-recommends \
42+
build-essential \
43+
cmake \
44+
git \
45+
libglib2.0-0 \
46+
ncurses-term \
47+
wget && \
48+
apt -y autoremove && apt clean autoclean && \
49+
rm -rf /var/lib/apt/lists/*
50+
51+
# Detect Ubuntu version and install CUDA 12.8 via NVIDIA network repo (cuda-keyring)
52+
RUN set -euo pipefail && \
53+
. /etc/os-release && \
54+
case "$ID" in \
55+
ubuntu) \
56+
case "$VERSION_ID" in \
57+
"20.04") cuda_repo="ubuntu2004";; \
58+
"22.04") cuda_repo="ubuntu2204";; \
59+
"24.04") cuda_repo="ubuntu2404";; \
60+
*) echo "Unsupported Ubuntu $VERSION_ID"; exit 1;; \
61+
esac ;; \
62+
*) echo "Unsupported base OS: $ID"; exit 1 ;; \
63+
esac && \
64+
apt-get update && apt-get install -y --no-install-recommends wget gnupg ca-certificates && \
65+
wget -q https://developer.download.nvidia.com/compute/cuda/repos/${cuda_repo}/x86_64/cuda-keyring_1.1-1_all.deb && \
66+
dpkg -i cuda-keyring_1.1-1_all.deb && \
67+
rm -f cuda-keyring_1.1-1_all.deb && \
68+
wget -q https://developer.download.nvidia.com/compute/cuda/repos/${cuda_repo}/x86_64/cuda-${cuda_repo}.pin && \
69+
mv cuda-${cuda_repo}.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
70+
apt-get update && \
71+
apt-get install -y --no-install-recommends cuda-toolkit-12-8 && \
72+
apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
73+
74+
75+
ENV CUDA_HOME=/usr/local/cuda-12.8
76+
ENV PATH=${CUDA_HOME}/bin:${PATH}
77+
ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}
78+
ENV TORCH_CUDA_ARCH_LIST=8.0+PTX
79+
80+
# Copy the Isaac Lab directory (files to exclude are defined in .dockerignore)
81+
COPY ../ ${ISAACLAB_PATH}
82+
83+
# Ensure isaaclab.sh has execute permissions
84+
RUN chmod +x ${ISAACLAB_PATH}/isaaclab.sh
85+
86+
# Set up a symbolic link between the installed Isaac Sim root folder and _isaac_sim in the Isaac Lab directory
87+
RUN ln -sf ${ISAACSIM_ROOT_PATH} ${ISAACLAB_PATH}/_isaac_sim
88+
89+
# Install toml dependency
90+
RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install toml
91+
92+
# Install apt dependencies for extensions that declare them in their extension.toml
93+
RUN --mount=type=cache,target=/var/cache/apt \
94+
${ISAACLAB_PATH}/isaaclab.sh -p ${ISAACLAB_PATH}/tools/install_deps.py apt ${ISAACLAB_PATH}/source && \
95+
apt -y autoremove && apt clean autoclean && \
96+
rm -rf /var/lib/apt/lists/*
97+
98+
# for singularity usage, have to create the directories that will binded
99+
RUN mkdir -p ${ISAACSIM_ROOT_PATH}/kit/cache && \
100+
mkdir -p ${DOCKER_USER_HOME}/.cache/ov && \
101+
mkdir -p ${DOCKER_USER_HOME}/.cache/pip && \
102+
mkdir -p ${DOCKER_USER_HOME}/.cache/nvidia/GLCache && \
103+
mkdir -p ${DOCKER_USER_HOME}/.nv/ComputeCache && \
104+
mkdir -p ${DOCKER_USER_HOME}/.nvidia-omniverse/logs && \
105+
mkdir -p ${DOCKER_USER_HOME}/.local/share/ov/data && \
106+
mkdir -p ${DOCKER_USER_HOME}/Documents
107+
108+
# for singularity usage, create NVIDIA binary placeholders
109+
RUN touch /bin/nvidia-smi && \
110+
touch /bin/nvidia-debugdump && \
111+
touch /bin/nvidia-persistenced && \
112+
touch /bin/nvidia-cuda-mps-control && \
113+
touch /bin/nvidia-cuda-mps-server && \
114+
touch /etc/localtime && \
115+
mkdir -p /var/run/nvidia-persistenced && \
116+
touch /var/run/nvidia-persistenced/socket
117+
118+
# installing Isaac Lab dependencies
119+
# use pip caching to avoid reinstalling large packages
120+
RUN --mount=type=cache,target=${DOCKER_USER_HOME}/.cache/pip \
121+
${ISAACLAB_PATH}/isaaclab.sh --install
122+
123+
# Install cuRobo from source (pinned commit); needs CUDA env and Torch
124+
RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation \
125+
"nvidia-curobo @ git+https://github.com/NVlabs/curobo.git@ebb71702f3f70e767f40fd8e050674af0288abe8"
126+
127+
# HACK: Remove install of quadprog dependency
128+
RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip uninstall -y quadprog
129+
130+
# aliasing isaaclab.sh and python for convenience
131+
RUN echo "export ISAACLAB_PATH=${ISAACLAB_PATH}" >> ${HOME}/.bashrc && \
132+
echo "alias isaaclab=${ISAACLAB_PATH}/isaaclab.sh" >> ${HOME}/.bashrc && \
133+
echo "alias python=${ISAACLAB_PATH}/_isaac_sim/python.sh" >> ${HOME}/.bashrc && \
134+
echo "alias python3=${ISAACLAB_PATH}/_isaac_sim/python.sh" >> ${HOME}/.bashrc && \
135+
echo "alias pip='${ISAACLAB_PATH}/_isaac_sim/python.sh -m pip'" >> ${HOME}/.bashrc && \
136+
echo "alias pip3='${ISAACLAB_PATH}/_isaac_sim/python.sh -m pip'" >> ${HOME}/.bashrc && \
137+
echo "alias tensorboard='${ISAACLAB_PATH}/_isaac_sim/python.sh ${ISAACLAB_PATH}/_isaac_sim/tensorboard'" >> ${HOME}/.bashrc && \
138+
echo "export TZ=$(date +%Z)" >> ${HOME}/.bashrc && \
139+
echo "shopt -s histappend" >> /root/.bashrc && \
140+
echo "PROMPT_COMMAND='history -a'" >> /root/.bashrc
141+
142+
# make working directory as the Isaac Lab directory
143+
# this is the default directory when the container is run
144+
WORKDIR ${ISAACLAB_PATH}

0 commit comments

Comments
 (0)