Conversation
There was a problem hiding this comment.
Pull Request Overview
Adds remote Rerun Viewer support and compresses image logging to reduce bandwidth when running headless.
- Add display_url and display_port to TeleoperateConfig and RecordConfig and pass them to init_rerun.
- Switch image logging from rr.Image to JPEG via OpenCV and rr.ImageEncoded.
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| src/lerobot/utils/visualization_utils.py | Adds remote connection in init_rerun and JPEG encoding in log_rerun_data. |
| src/lerobot/scripts/lerobot_teleoperate.py | Adds display_url and display_port to TeleoperateConfig and passes them to init_rerun. |
| src/lerobot/scripts/lerobot_record.py | Adds display_url and display_port to RecordConfig and passes them to init_rerun. |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| memory_limit = os.getenv("LEROBOT_RERUN_MEMORY_LIMIT", "10%") | ||
| rr.spawn(memory_limit=memory_limit) | ||
| if url: | ||
| rr.connect_grpc(url=f"rerun+http://{url}:{port}/proxy") |
There was a problem hiding this comment.
The Python SDK exposes rr.connect, not rr.connect_grpc; this will raise an AttributeError. Use rr.connect with host:port (e.g., rr.connect(f"{url}:{port}")) rather than a rerun+http URL with /proxy.
| rr.connect_grpc(url=f"rerun+http://{url}:{port}/proxy") | |
| rr.connect(f"{url}:{port}") |
| ".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50] | ||
| ) | ||
| encoded_image = buffer.tobytes() | ||
| rr.log(key, rr.EncodedImage(contents=encoded_image, media_type="image/jpeg"), static=True) |
There was a problem hiding this comment.
The logged type should be rr.ImageEncoded, not rr.EncodedImage; rr.EncodedImage does not exist in the Python API and will fail at runtime. Replace with rr.ImageEncoded(contents=..., media_type="image/jpeg").
| rr.log(key, rr.EncodedImage(contents=encoded_image, media_type="image/jpeg"), static=True) | |
| rr.log(key, rr.ImageEncoded(contents=encoded_image, media_type="image/jpeg"), static=True) |
| _, buffer = cv2.imencode( | ||
| ".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50] |
There was a problem hiding this comment.
This path assumes 3-channel RGB input; it will fail or produce incorrect results for 1-channel (grayscale) and 4-channel (RGBA) images mentioned in the docstring. Handle channel count explicitly: no color conversion for single-channel, RGB→BGR for 3-channel, and RGBA→BGR (drop alpha) for 4-channel before JPEG encoding.
| _, buffer = cv2.imencode( | |
| ".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50] | |
| # Handle channel count explicitly for JPEG encoding | |
| if arr.ndim == 3: | |
| if arr.shape[2] == 1: | |
| # Grayscale, no color conversion needed | |
| arr_to_encode = arr | |
| elif arr.shape[2] == 3: | |
| # RGB to BGR | |
| arr_to_encode = cv2.cvtColor(arr, cv2.COLOR_RGB2BGR) | |
| elif arr.shape[2] == 4: | |
| # RGBA to BGR (drop alpha) | |
| arr_to_encode = cv2.cvtColor(arr[:, :, :3], cv2.COLOR_RGB2BGR) | |
| else: | |
| raise ValueError(f"Unsupported number of channels for image encoding: {arr.shape[2]}") | |
| else: | |
| raise ValueError(f"Expected 3D array for image encoding, got shape {arr.shape}") | |
| _, buffer = cv2.imencode( | |
| ".jpg", arr_to_encode, [int(cv2.IMWRITE_JPEG_QUALITY), 50] |
| _, buffer = cv2.imencode( | ||
| ".jpg", cv2.cvtColor(arr, cv2.COLOR_RGB2BGR), [int(cv2.IMWRITE_JPEG_QUALITY), 50] | ||
| ) |
There was a problem hiding this comment.
[nitpick] cv2.imencode expects uint8 data; if arr is float (e.g., [0,1]) or another dtype, encoding may fail or yield incorrect output. Consider converting/scaling to uint8 prior to encoding and checking the boolean return value of cv2.imencode to handle failures.
| import os | ||
| from typing import Any | ||
|
|
||
| import cv2 |
There was a problem hiding this comment.
[nitpick] Importing OpenCV at module import time forces a heavy optional dependency even when visualization is disabled. Move the import inside log_rerun_data (or behind the JPEG branch) to avoid unnecessary runtime/import errors on setups without OpenCV.
| # Display all cameras on screen | ||
| display_data: bool = False | ||
| # Display data on a remote Rerun server | ||
| display_url: str = None |
There was a problem hiding this comment.
[nitpick] Type annotation for display_url is inconsistent with the rest of the config (which uses the PEP 604 style, e.g., float | None). For consistency, use str | None for display_url.
| display_url: str = None | |
| display_url: str | None = None |
| # Display all cameras on screen | ||
| display_data: bool = False | ||
| # Display data on a remote Rerun server | ||
| display_url: str = None |
There was a problem hiding this comment.
[nitpick] Align the type of display_url with the project's union style by using str | None for consistency with other fields (e.g., teleop_time_s: float | None).
| display_url: str = None | |
| display_url: str | None = None |
pkooij
added
the
visualization
|
@pkooij Do you have any idea when this might get merged? It would be very useful for me. I assume @imstevenpmwork may not have time to review at the moment, so maybe somebody else with permissions could take a look? Thank you for contributing @J4nn1K! |
|
Hello everyone, I'm currently taking a look 😄 Thanks for pointing me to this and for the contribution! |
|
Superseded by: #2756 I will add you as a co-authors. It would be great if someone can test it in a remote or headless setup 😄 |
|
Implemented in: #2767 |
|
Thank you @imstevenpmwork! |
4 participants
What this does
When running LeRobot headless (on a NVIDIA Jetson for example), it can be helpful to visualize data on a remote machine. This PR changes two things:
TeleoperateConfigandRecordConfig: (1)display_urland (2)display_port. Whendisplay_urlis set, the Rerun SDK will connects to a remote viewer and stream all the data via gRPC. Read more about Rerun's operating modes here.rr.Imageto compressed JPEG viarr.EncodedImageto save space and bandwidth.How it was tested
Tested buy running multiple teleop and recording trials on a Jetson AGX Thor with data streamed over a VPN to different devices like: Raspberry Pi, Jetson Orin Nano, Macbook.
How to checkout & try?
Start a Rerun Viewer on a remote machine
Start a teleoperation loop:
lerobot-teleoperate \ --robot.type=YOUR_ROBOT \ --robot.port=YOUR_PORT \ --robot.cameras="main: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}" --teleop.type=YOUR_ROBOT \ --teleop.port=YOUR_PORT \ --display_data=true \ --display_url=YOUR_REMOTE_IPThe remote viewer should start displaying the data.