RAI

Important

RAI is meant for R&D. Make sure to understand its limitations.

RAI is a flexible AI agent framework to develop and deploy Embodied AI features for your robots.

---

---

---

Overview

The RAI framework aims to:

• Supply a general multi-agent system, bringing Gen AI features to your robots.
• Add human interactivity, flexibility in problem-solving, and out-of-box AI features to existing robot stacks.
• Provide first-class support for multi-modalities, enabling interaction with various data types.

Limitations

• Limitations of LLMs and VLMs in use apply: poor spatial reasoning, hallucinations, jailbreaks, latencies, costs, ...
• Resource use (memory, CPU) is not addressed yet.​
• Requires connectivity and / or an edge platform.​

Table of Contents

• Features
• Setup
• Usage examples (demos)
• Debugging Assistant
• Developer resources

Features

• Voice interaction (both ways).
• Customizable robot identity, including constitution (ethical code) and documentation (understanding own capabilities).
• Accessing camera ("What do you see?"), utilizing VLMs.
• Summarizing own state through ROS logs.
• ROS 2 action calling and other interfaces. The Agent can dynamically list interfaces, check their message type, and publish.
• Integration with LangChain to abstract vendors and access convenient AI tools.
• Tasks in natural language to nav2 goals.
• NoMaD integration.
• Tracing.
• Grounded SAM 2 integration.
• Improved Human-Robot Interaction with voice and text.
• Additional tooling such as GroundingDino.
• Support for at least 3 different AI vendors.
• Debugging assistant for ROS 2.
• SDK for RAI developers.
• UI for configuration to select features and tools relevant for your deployment.

Setup

Before going further, make sure you have ROS 2 (Jazzy or Humble) installed and sourced on your system.

1. Setting up the workspace:

1.1 Install poetry

RAI uses Poetry for python packaging and dependency management. Install poetry (1.8+) with the following line:

curl -sSL https://install.python-poetry.org | python3 -

Alternatively, you can opt to do so by following the official docs.

1.2 Clone the repository:

git clone https://github.com/RobotecAI/rai.git
cd rai

1.3 Create poetry virtual environment and install dependencies:

poetry install
rosdep install --from-paths src --ignore-src -r -y

Tip

If you want to use features such as Grounded SAM 2 or NoMaD install additional dependencies:

poetry install --with openset,nomad

1.4 Configure RAI

Run the configuration tool to set up your vendor and other settings:

poetry shell
streamlit run src/rai/rai/utils/configurator.py

Tip

If the web browser does not open automatically, open the URL displayed in the terminal manually.

2. Build the project:

2.1 Build RAI workspace

colcon build --symlink-install

2.2 Activate a virtual environment:

source ./setup_shell.sh

3. Setting up vendors

RAI is vendor-agnostic. Use the configuration in config.toml to set up your vendor of choice for RAI modules. Vendor choices for RAI and our recommendations are summarized in Vendors Overview.

We strongly recommend you to use of best-performing AI models to get the most out of RAI!

Pick your local solution or service provider and follow one of these guides:

• Ollama
• OpenAI
• AWS Bedrock

What's next?

Once you know your way around RAI, try the following challenges, with the aid the developer guide:

• Run RAI on your own robot and talk to it, asking questions about what is in its documentation (and others!).
• Implement additional tools and use them in your interaction.
• Try a complex, multi-step task for your robot, such as going to several points to perform observations!

Debugging Assistant

Use the debugging assistant to inspect ROS 2 network state and troubleshoot issues.

Simulation demos

Try RAI yourself with these demos:

Application	Robot	Description	Docs Link
Mission and obstacle reasoning in orchards	Autonomous tractor	In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation.	link
Manipulation tasks with natural language	Robot Arm (Franka Panda)	Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2	link
Autonomous mobile robot demo	Husarion ROSbot XL	Demonstrate RAI's interaction with an autonomous mobile robot platform for navigation and control	link
Turtlebot demo	Turtlebot	Showcase RAI's capabilities with the popular Turtlebot platform	link
Speech-to-speech interaction with autonomous taxi	Simulated car	Demonstrate RAI's speech-to-speech interaction capabilities for specifying destinations to an autonomous taxi in awsim with autoware environment	link

Community

Embodied AI Community Group

RAI is one of the main projects in focus of the Embodied AI Community Group. If you would like to join the next meeting, look for it in the ROS Community Calendar.

Publicity

• A talk about RAI at ROSCon 2024.

RAI Q&A

Please take a look at Q&A.

Developer Resources

See our Developer Guide for a deeper dive into RAI, including instructions on creating a configuration specifically for your robot.

Contributing

You are welcome to contribute to RAI! Please see our Contribution Guide.