Improved native ROS support

[jleibs]

This is a tracking/discussion topic for future ROS support.

The Current State

We currently bridge into ROS by creating a node which re-logs specific topics to rerun equivalents. The ROS how-to guide covers what's involved in making this work.

This enables a motivated user to interoperate with Rerun, but requires writing a fair amount of custom code and is not great from a performance perspective.

Open Issues that would improve ROS support:

• Allow Transformations to be aware of a "fixed frame" #1522
• Log an image explicitly as a 3D surface/billboard #1531
• Support for "Derived" Transforms #1533
• Support for LaserScan projection models #1534
• Rotate image views in viewer #3982

Future Possibilities

Generic Rerun Node

The easiest next step would be following the same approach, but creating a dedicated generic / configurable Rerun bridge. This would auto-detect all topics, and create appropriate loggers for everything (with some configurable / wild-card style options for filtering or remapping topics -> paths). "Known messages" could be converted to Rerun types. Unknown messages could still be logged as extension components. The largest obstacle to this is #1533, at which point the mapping ros-msg -> entity event becomes much more straight-forward.

While it would certainly be fastest to prototype this in Python, this could be a good opportunity to investigate https://github.com/ros2-rust/ros2_rust , which would probably have significant performance benefits (important if we are trying to process all the messages from a ROS system), and also set us up better for future steps since our short-term plugin story is likely to be rust-centric.

Generic Rosbag-loader

Similar to the above, it would be nice to implement the same functionality but going direct from a a Rosbag or MCAP file without needing to connect to a ROS runtime system at all. The message-processing and topic mapping, however, would otherwise be identical.

Viewer-side ROS-msg handlers

Building on top of a more generic Rerun node the next step would be to do more handling of ROS messages directly in the store/viewer. ROS messages could be logged with ROS-specific component types (e.g. ros2.sensor_msgs.msg.Image), and then a Viewer-plugin could convert them to Rerun-native primitives as a derived component. This would allow us to relay all messages fully generically into the store without ever deserializing the payload, vastly simplifying the bridge code.

Viewer-side plugin for receiving messages

Once we have viewer-native support for ROS-serialized components, it would be nice to remove the need for having a separate standalone Rerun-ROS bridge node all together. A viewer-plugin that acted like a ROS node could in theory inject messages directly into the store without needing to go through a separate log step, removing one more copy from the pipeline.

At this point the Rerun viewer (with the ROS plugin enabled) would be able to handle many of the simpler Rviz use-cases while simultaneously still supporting libraries that do ROS-free rerun logging, bringing all of the data into a single coherent view.

[AlexKaravaev]

"creating a dedicated generic / configurable Rerun bridge" - is there already a repo/branch that is tackling that?

Will be willing to contribute! Found this https://github.com/rerun-io/cpp-example-ros-bridge which logically actually implements a generic bridge, but example in the name confuses me 😄 .

Tagging @roym899 for visibility

[roym899]

Yes, that's the most generic approach right now. It's currently more of a proof-of-concept than a fully featured ROS bridge (hence the example in the name).

PRs to support more standard types are definitely welcome.

There's also a ROS 2 version in the works and I'll plan to keep features in sync once the basics work well.

[yukkysaito]

I am very excited about this project. We have been looking for better visualization tools for Autoware, and I am very interested in this project.
https://github.com/autowarefoundation/autoware

[karimo87]

A quick update: we're working on a production grade ROS bridge in Rust over here. We're working hard to get it ready to handle more contributors within the next weeks, but we're not there yet.