Adds SkillGen framework to Isaac Lab with cuRobo support #3303

njawale42 · 2025-08-28T18:32:18Z

Description

This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility.

Note:

Please look at the cuRobo usage license
Please look at updated isaacsim license

Technical Implementation:

Annotation Framework:

Manual subtask start annotations to cleanly separate skill execution from motion-planning segments
Consistent trajectory segmentation for downstream dataset consumers

Motion Planning:

Base Motion Planner (Extensible):
- Introduces a reusable planner interface for uniform integration:
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py
- Defines a minimal, consistent API for planners:
  - update_world_and_plan_motion(...), get_planned_poses(...), etc.
- The cuRobo planner inherits from this base class.
- New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline.
CuRobo Planner (GPU-accelerated, collision-aware):
- Multi-phase planning: approach → contact → retreat
- Dynamic object attach/detach and contact-aware sphere management
- Real-time world synchronization between Isaac Lab and cuRobo
- Configurable collision filtering for contact phases
- Tests:
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py
  - source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py
  - source/isaaclab_mimic/test/test_generate_dataset_skillgen.py

Data Generation Pipeline:

Automated dataset generation with precise skill-based segmentation
Integrates with existing observation/action spaces
Supports multi-env parallel collection with cuRobo-backed planning

Visualization and Debugging:

Rerun-based 3D visualization for trajectory/collision inspection
Real-time sphere visualization for collision boundaries and contact phases

Dependencies:

cuRobo: motion planning and collision checking
Rerun: 3D visualization and debugging

Integration:

This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users.

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

Screenshot

SkillGen Data Generation

Cube Stacking SkillGen Data Generation	Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)

Bin Cube Stacking Behavior Cloned Policy

Rerun Integration

Motion Planner Tests

Obstacle Avoidance (cuRobo)	Cube Stack End-to-End (cuRobo)

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

gavrielstate · 2025-08-28T21:00:46Z

README.md


 The Isaac Lab framework is released under [BSD-3 License](LICENSE). The `isaaclab_mimic` extension and its corresponding standalone scripts are released under [Apache 2.0](LICENSE-mimic). The license files of its dependencies and assets are present in the [`docs/licenses`](docs/licenses) directory.

+Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE) for information on Isaac Sim licensing.


This should just point to the docs/licenses/dependencies/isaacsim-license.txt license file, which should be updated with the contents of https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE

Corrected! Thanks Gav!

scripts/imitation_learning/isaaclab_mimic/annotate_demos.py

scripts/imitation_learning/isaaclab_mimic/generate_dataset.py

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info.py

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info_pool.py

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/plan_visualizer.py

.../isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py

peterd-NV

Tested/verified skillgen/prior mimic workflows working as expected. My prior review was completed in IsaacLab-Internal repo and all comments have been addressed in this public PR.

njawale42 · 2025-09-03T18:20:10Z

@nv-caelan FYI I included most of your suggested changes in my commits (did not apply directly through github).

kellyguo11

is there any documentation / tutorial available for this? including installation steps and how to set it up to use skillgen, what it does, how it works, the visualizer...

scripts/imitation_learning/isaaclab_mimic/generate_dataset.py

source/isaaclab_mimic/isaaclab_mimic/datagen/data_generator.py

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info_pool.py

kellyguo11 · 2025-09-03T21:36:52Z

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/curobo_planner.py

+from dataclasses import dataclass
+from typing import Any
+
+from curobo.cuda_robot_model.cuda_robot_model import CudaRobotModelState


will curobo be a hard dependency for all mimic workflows? will this cause any issues if curobo is not installed?

With use_skillgen=False, normal Mimic runs unchanged: generate_dataset.py doesn’t import or construct the cuRobo planner, so no cuRobo code is touched. Will make sure to test this with and without cuRobo once.

sounds good, please verify if things still run without curobo dependency installed

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/curobo_planner.py

kellyguo11 · 2025-09-03T22:28:42Z

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/plan_visualizer.py

+import weakref
+from typing import TYPE_CHECKING, Any, Optional
+
+import rerun as rr


does this need to be added as a dependency?

Yes, however this file is only initialized if visualize_plan is set to True in curobo config for the particular task. So this will only be used then and people can run this without rerun if they set the flag to false.

are we going to include it in setup.py? or ask users to install it independently?

up to us, i think that since we ask users to install curobo independently from source, we can also add pip install for rerun conditional on this flag in the documentation. what do you think?

if it's not needed for other workflows, we can ask users to install it. let's also add a check in the code if it wasn't install and hint users to install the dependencies needed

Yes, will add a try-except while importing rerun in plan_visualizer.py and raise an ImportError if it isn't installed.

ooctipus · 2025-09-04T03:00:36Z

source/isaaclab_mimic/isaaclab_mimic/envs/__init__.py

+    id="Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0",
+    entry_point="isaaclab_mimic.envs:FrankaCubeStackIKRelMimicEnv",
+    kwargs={
+        "env_cfg_entry_point": franka_stack_ik_rel_skillgen_env_cfg.FrankaCubeStackIKRelSkillgenEnvCfg,


Not for this PR maybe, but we should be moving toward using string typed entry_point registration rather than import.

is there anything preventing us from using the same mechanism as the RL entry points in this PR?

I can take this up in a separate PR as well (can be tried for all mimic workflows)

...sks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/stack_joint_pos_env_cfg.py

source/isaaclab/isaaclab/envs/manager_based_rl_mimic_env.py

ooctipus

I am not super familiar with mimic flow and the my first pass looks good, it will be nice if have mimic folks provide more detail view.

My have only 2 comments:

it will be great if we don't touch os env_variable.
to me mimic_env_cfg and manager_based_rl_mimic_env seems to diverge further and further, and maybe more appropriate to go isaaclab_mimic.envs since there is also envs folder there. You will have more separation of concern and flexibility to tailor for mimic related env workflow and entrypoint in the end.

nv-caelan

LGTM, thanks for addressing my comments!

docker/Dockerfile.base

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py

Mayankm96 · 2025-09-05T09:43:04Z

docs/licenses/dependencies/isaacsim-license.txt

@@ -1,13 +1,188 @@
-Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
+The Isaac Sim software in this repository is covered under the Apache 2.0


Can we please move this license change into a separate MR as it has nothing to do with CuRobo.

I added these as per @gavrielstate 's instructions.
@kellyguo11 for viz

I agree would be good to have a separate PR. let me just create a new PR for the licensing updates.

Mayankm96 · 2025-09-05T09:45:21Z

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info.py

        eef_pose=None,
        object_poses=None,
        subtask_term_signals=None,
+        subtask_start_signals=None,


Maybe this is coming from legacy code but we want type-hinting to live on function level and not docstrings. Please check here for style-guide: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html#type-hinting

We can change these (since I see such type hinting in quite a lot of scripts) in a separate MR?

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/curobo_planner_config.py

…or changes in cuRobo. Added urdf extraction and on-the-fly config creation from nucleus.

…emoving redundancies

…Robo installation in Dockerfile.base for CL runner to install cuRobo for tests

…cing a new config

…minor docstring changes

…at it was

Co-authored-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]>

Signed-off-by: Kelly Guo <[email protected]>

…solved doc compilation errors

Signed-off-by: Kelly Guo <[email protected]>

source/isaaclab_mimic/docs/CHANGELOG.rst

Signed-off-by: Kelly Guo <[email protected]>

) ## Description This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility. **Note:** - Please look at the cuRobo usage license ![here](docs/licenses/dependencies/cuRobo-license.txt) - Please look at updated isaacsim license ![here](docs/licenses/dependencies/isaacsim-license.txt) ### Technical Implementation: **Annotation Framework:** - Manual subtask start annotations to cleanly separate skill execution from motion-planning segments - Consistent trajectory segmentation for downstream dataset consumers **Motion Planning:** - **Base Motion Planner (Extensible):** - Introduces a reusable planner interface for uniform integration: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py` - Defines a minimal, consistent API for planners: - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc. - The cuRobo planner inherits from this base class. - New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline. - **CuRobo Planner** (GPU-accelerated, collision-aware): - Multi-phase planning: approach → contact → retreat - Dynamic object attach/detach and contact-aware sphere management - Real-time world synchronization between Isaac Lab and cuRobo - Configurable collision filtering for contact phases - **Tests**: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py` - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py` - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py` **Data Generation Pipeline:** - Automated dataset generation with precise skill-based segmentation - Integrates with existing observation/action spaces - Supports multi-env parallel collection with cuRobo-backed planning **Visualization and Debugging:** - Rerun-based 3D visualization for trajectory/collision inspection - Real-time sphere visualization for collision boundaries and contact phases ### Dependencies: - **cuRobo**: motion planning and collision checking - **Rerun**: 3D visualization and debugging ### Integration: This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users. ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] This change requires a documentation update ## Screenshot ### SkillGen Data Generation <table> <tr> <td align="center"><strong>Cube Stacking SkillGen Data Generation</strong></td> <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae" alt="Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e" alt="Bin Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ### Bin Cube Stacking Behavior Cloned Policy ![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9) ### Rerun Integration ![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0) ### Motion Planner Tests <table> <tr> <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td> <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db" alt="Obstacle Avoidance cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde" alt="Cube Stack End-to-End cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Kelly Guo <[email protected]> Co-authored-by: Kelly Guo <[email protected]> Co-authored-by: Pascal Roth <[email protected]>

njawale42 requested review from Mayankm96, jtigue-bdai, kellyguo11 and ooctipus as code owners August 28, 2025 18:32

kellyguo11 requested a review from peterd-NV August 28, 2025 19:46

njawale42 requested review from peterd-NV and removed request for peterd-NV August 28, 2025 19:59

gavrielstate reviewed Aug 28, 2025

View reviewed changes

nv-caelan suggested changes Aug 28, 2025

View reviewed changes

njawale42 requested a review from nv-caelan August 28, 2025 22:36

peterd-NV approved these changes Aug 29, 2025

View reviewed changes

kellyguo11 changed the title ~~SkillGen in Isaac Lab for 2.3 Release~~ Adds SkillGen framework to Isaac Lab with cuRobo support Sep 3, 2025

kellyguo11 reviewed Sep 3, 2025

View reviewed changes

njawale42 requested review from hhansen-bdai and pascal-roth as code owners September 4, 2025 00:14

ooctipus reviewed Sep 4, 2025

View reviewed changes

...sks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/stack_joint_pos_env_cfg.py Outdated Show resolved Hide resolved

ooctipus reviewed Sep 4, 2025

View reviewed changes

source/isaaclab/isaaclab/envs/manager_based_rl_mimic_env.py Show resolved Hide resolved

ooctipus approved these changes Sep 4, 2025

View reviewed changes

nv-caelan approved these changes Sep 4, 2025

View reviewed changes

njawale42 removed the request for review from hhansen-bdai September 4, 2025 16:58

njawale42 force-pushed the neel/skillgen branch from cdd6d92 to ea7da61 Compare September 4, 2025 21:51

njawale42 removed request for Mayankm96 and pascal-roth September 4, 2025 22:46

kellyguo11 reviewed Sep 4, 2025

View reviewed changes

docker/Dockerfile.base Outdated Show resolved Hide resolved

Mayankm96 reviewed Sep 5, 2025

View reviewed changes

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py Show resolved Hide resolved

Mayankm96 reviewed Sep 5, 2025

View reviewed changes

source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/curobo_planner_config.py Outdated Show resolved Hide resolved

njawale42 added 18 commits September 8, 2025 15:38

Updated cuRobo and isaac sim licenses. Fixed rerun path viz bugs. Min…

5ccb27f

…or changes in cuRobo. Added urdf extraction and on-the-fly config creation from nucleus.

Added data gen test for skillgen, and changes to planner and visualizer

7d1c284

Minor change in README regarding isaacsim license file

048ab03

Addressing review comments: type hints, conditional simplification, r…

beaeaa9

…emoving redundancies

Added make pose helper, minor code refactor for cuRobo planner

a2e64cd

Made rerun import optional depending on visualize_plan flag, added cu…

aed7dcb

…Robo installation in Dockerfile.base for CL runner to install cuRobo for tests

Simplified skillgen related offset and object pose changes by introdu…

025f15e

…cing a new config

Added lazy logging to avoid using debug branching in planner script, …

6a91163

…minor docstring changes

Added docstring for the logger class

52c5e32

Added a new dockerfile for curobo, reverted the base dockerfile to wh…

5234119

…at it was

Added first version of the skillgen.rst doc, minor config change

f016d58

Addressing Mayank's comments, Documentation changes

76b9d95

Small bug fix

7dd4e26

Doc update, cuRobo tests teardown update

c3226cb

Changed curobo config to use configclass instead of dataclass

5184660

Documentation updates, minor code updates

c6716a7

Added task gifs to the docs

253e506

Resolve linter errors

af2c7ff

njawale42 force-pushed the neel/skillgen branch from 12ae4bb to af2c7ff Compare September 8, 2025 22:40

kellyguo11 and others added 6 commits September 8, 2025 17:35

Update source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info_pool.py

a4c8187

Co-authored-by: Pascal Roth <[email protected]> Signed-off-by: Kelly Guo <[email protected]>

Merge branch 'main' into neel/skillgen

4b5a2c1

Signed-off-by: Kelly Guo <[email protected]>

Added updates in changelogs for isaaclab_mimic and isaaclab_tasks. Re…

31af671

…solved doc compilation errors

Merge branch 'main' into neel/skillgen

3d6d2ee

Signed-off-by: Kelly Guo <[email protected]>

Update CHANGELOG.rst

56f863c

Signed-off-by: Kelly Guo <[email protected]>

Update extension.toml

fbb4605

Signed-off-by: Kelly Guo <[email protected]>

kellyguo11 reviewed Sep 9, 2025

View reviewed changes

source/isaaclab_mimic/docs/CHANGELOG.rst Show resolved Hide resolved

kellyguo11 added 2 commits September 8, 2025 21:30

Update source/isaaclab_mimic/docs/CHANGELOG.rst

1fd97bb

Signed-off-by: Kelly Guo <[email protected]>

Update __init__.py

3cee172

Signed-off-by: Kelly Guo <[email protected]>

kellyguo11 merged commit 994979c into isaac-sim:main Sep 9, 2025
6 checks passed


		The Isaac Lab framework is released under [BSD-3 License](LICENSE). The `isaaclab_mimic` extension and its corresponding standalone scripts are released under [Apache 2.0](LICENSE-mimic). The license files of its dependencies and assets are present in the [`docs/licenses`](docs/licenses) directory.

		Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](https://github.com/isaac-sim/IsaacSim/blob/main/LICENSE) for information on Isaac Sim licensing.

		@@ -1,13 +1,188 @@
		Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
		The Isaac Sim software in this repository is covered under the Apache 2.0

Adds SkillGen framework to Isaac Lab with cuRobo support #3303

Adds SkillGen framework to Isaac Lab with cuRobo support #3303

Uh oh!

Conversation

njawale42 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Technical Implementation:

Dependencies:

Integration:

Type of change

Screenshot

SkillGen Data Generation

Bin Cube Stacking Behavior Cloned Policy

Rerun Integration

Motion Planner Tests

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

peterd-NV left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njawale42 commented Sep 3, 2025

Uh oh!

kellyguo11 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njawale42 Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

njawale42 Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ooctipus left a comment

njawale42 commented Aug 28, 2025 •

edited

Loading

peterd-NV left a comment •

edited

Loading

njawale42 Sep 4, 2025 •

edited

Loading

njawale42 Sep 7, 2025 •

edited

Loading