Add support for rerun visualizer

[jviereck]

rerun.io is a new tool for plotting and visualizing time indexed data. It would be great if similar to the meshcat viewer, pinocchio would ship with a rerun viewer out of the box as well.

[jviereck]

I have python code outside of pinocchio for this. If the maintainer think adding rerun support to pinocchio is good, I can work on a PR.

\cc @wxmerkt @jcarpent

[ManifoldFR]

I recall there are people at LAAS working on a Python rerun visualizer (with some Rust plugins for the client I think?)

Is that right @nim65s ?

[olivier-stasse]

FYI https://github.com/Gepetto/gepetto-viewer-rerun

[jcarpent]

rerun.io is a new tool for plotting and visualizing time indexed data. It would be great if similar to the meshcat viewer, pinocchio would ship with a rerun viewer out of the box as well.

Dear Julian,

Excellent idea. Together with other members of Willow, we would be more than happy to discuss this initiative. It makes a while that we are thinking to add new visualization features to Pinocchio, up-to-date compared to Meshcat and others.
Could you send me an email to set-up a meeting?

Best,
Justin

[florent-lamiraux]

Indeed, we tried using rerun to display pinocchio models, but the outcome is a little bit disappointing. The tool has actually not been designed to display 3D shapes and the student who worked on the project contributed more on rerun itself than on displaying pinocchio models. There are several shortcomings, the most annoying one is that there is no API to access objects in the model tree, therefore it is difficult for example to make collision geometry invisible at loading.

If you think it is useful, I can participate to the meeting suggested by @jcarpent.

[jviereck]

Rerun has it's short comings at the moment. I worked on their codebase and adding a toggle to turn meshes on and off shouldn't be hard to implement. I also know the rerun team is looking to hire a person that is dedicated for adding more robotics centric functionality to rerun. So if we meet and come up with a list of things that would help us, then they might be happy to take it up and implement it.

[jviereck]

Just to share how we managed to use rerun with pinocchio:

• Have a set of .obj files for basic shapes like cylinders, plains and circles of unit size 1 m. Apply a scaling to them to get them to the desired size.
• To load other meshes, I load the meshes in Python (meshio can load many mesh formats) and send the point and triangles from python to rerun.
• To update the mesh locations, I log transformations to rerun.

The setup is rather simple and doesn't need any change / setup to rerun.

Would this be a way to add support for rerun to pinocchio? I can put the code I have in a pull request in this repo.

[ManifoldFR]

@jviereck this sounds like what I've done on pinocchio-rerun which is written in C++ (using rerun's C++ API). On there I use assimp to load all kinds of mesh formats without trouble, then load the vertex positions, normals, other attributes, and face indices into the rerun-cpp Mesh3D archetype, which I can log/upload to the viewer.

Logging transformations is the right way to go IMO, that's how the meshcat viewer is implemented, same goes for just about any 3D viz stuff I think.

[jviereck]

@ManifoldFR great to hear you followed a similar approach. Any reason you went for C++? Do you think implementing things in Python like the MeshcatViewer right now in Pinocchio is good enough?

[ManifoldFR]

@jviereck at that point all I wanted was something to visualize objects from C++ without having to go through Python.

[florent-lamiraux]

As mentioned above by @olivier-stasse, a student spent 4 months at LAAS developping a solution in rust: gepetto-viewer-rerun. We initially had a look at pinocchio-rerun. The main issues were :

• there was no collada parser in rerun,
• meshes are unicolor.
  This was a big limitation for robotics applications.
  The student - interacting with rerun developers - developed a collada parser and fixed some limitations in rerun itself:
• Allow overriding albedo color on Asset3D rerun-io/rerun#7458,
• Fix CpuMesh.albedo_factor overwriting rerun-io/rerun#7914.
  At the end of the internship, I had the impression that rerun was not really designed for the purpose of displaying 3D shapes.
  In any case, I think gepetto-viewer-rerun is the right starting point.

[jcarpent]

• In any case, I think gepetto-viewer-rerun is the right starting point.

I don't really agree on this point, as gepetto-viewer-rerun is fully Python-based, while pinocchio-rerun has Python bindings, meaning that it can be easily integrated inside a C++ pipeline with minimal effort.