Setup Orin & ZED Cameras

[jruths]

The new NVIDIA Orin needs to be set up and configured to aggregate images & depth images from the four ZED cameras. Stereolabs (ZED) has a ROS SDK for doing much of the calibration and aggregation from multiple cameras. We should be able to stitch together a birds-eye-view image (often used throughout the industry).

In terms of design, the Orin will receive all the ZED data - and possibly also the LIDAR and RADAR data, do any needed pre-processing (e.g., birds-eye-view image, ground plane removal for lidar, etc) and publish cleaned perception topics to the main computer. To do this, the Orin and main computer will need to be on the same local network to share ROS messages.

This task could be coordinated with a member of the HFE team or done entirely by one person. Either way there will be a lot of hands on.

[jruths]

Some related tasks mainly since the ZED SDK has built in ways to configure the 4 ZED cameras into a single 360-degree image and depth map. Then this depth map could be used and aligned with the LIDAR and RADAR point clouds to determine the transformations that align these with the depth map.

• Automatic Sensor Calibration
• Projecting images onto the pointcloud

[MtGuerenS]

12/3/2024 Update:

Status: Connecting ZED and ROS

Next Steps:

1. Setup Nvidia ORIN
   a. Connect the physical
   b. Factory reset the device
   c. Install JetPack Components
2. Complete the setup for ZED cameras
3. Test out ZED cameras on ORIN
4. Download docker on ORIN
5. Setup/Configure ROS2
6. Connect/Test ROS2 and ZED SDK
7. Connect ORIN to the vehicle local network
8. Test ROS2 messaging across the devices
9. Setup the ROS2 ZED nodes and messages
10. Mount the ZED cameras and ORIN
11. Get the ZED_Wrapper to start on ORIN power on

Update:

• Much has been done since the last update
• Created a launch file on the ORIN to run 4 cameras at the same time on different topics
  • Commented out unnecessary topics advertised through the ZED_Wrapper
  • Could not comment out unnecessary compression types since that is not created by the ZED team
• Tested different models, publishing rates, quality of images, and running cameras on threads/processes. Here are my findings:
  • When running 4 cameras at the lowest performance depth perception model, it was publishing at a rate of 5 Hz. See graph 1.
  • Using one camera and the performance model, the camera was publishing images at 14 Hz with a resolution of 600x1060. See graph 2.
  • So using 1 camera with ultra would produce stable and efficient results, but using 4 cameras at performance affected the rate at which all cameras were publishing images severely.
  • I was able to get all 4 cameras running at 15 Hz with a resolution of 900x1200, uping the quality by 50% from before.
    • With this setup most of the CPUs are sitting at about 80% capacity.
    • Might be able to use depth perception on one camera at quality lower than 600x1060 and manage to stabilize at 15 Hz.
      • Could be worth creating another task to check out.

• Figured out how to connect the On Logic and ORIN together
  • Used Local Link for both IPv4 and IPv6, set ROS_DOMAIN_ID=1
• Subscribing to the image topics across the network, which were publishing at 15 Hz, was only able to work at < 1 Hz
  • I figured it was some bandwidth issue with sending 60 900x1200p images every second
  • Subscribed to the compressed images stably at 15 Hz, needed to uncompress the images on the On Logic Computer
  • Created a launch file launch.zed_cameras.py and package zed_cameras
    • Running the launch file creates 4 processes (nodes) for each camera and they uncompress the images and publish them on the computer
    • My un-compression method was originally too slow and led to confusion while trying to figure out how to speed it up
      • Use imports instead of plain python
• Created a System Service for automatically running the launch file on start
  • The service file is stored in /etc/systemd/system
  • Had to make sure that the dependencies were correct
  • Ran into an issue:
    • I was getting no logs from the service on boot and when I logged in the service wasn't on
    • I fixed it by running sudo systemd disable SERVICE_NAME and sudo systemd enable SERVICE_NAME
      • It relinks the file because sometimes making edits can mess with that