Visualizer unexpectedly splits 3D space view into two sub-groups of images

[gokcergun]

Hi,

Currently, I'm using Rerun to visualize 16 pinhole cameras. When I run the script, sometimes it visualizes all 16 cameras together, sometimes it only visualizes the first 9 cameras and when I "reset viewer" using gui, it visualizes only the rest of the cameras (10-16). When I reset again, it visualizes the first 9 again, and is stuck between these two alternatives. Sometimes visualizing a totally different scene, and then visualizing the original (16-camera) scene helps to get rid of this bug and I am able to visualize 16 cameras correctly. Sometimes it does not help.

The related part of the code is following:

for camera_name in calibrations.keys():

        calibration = calibrations[camera_name]
        K = np.array(calibration['intrinsics_matrix'])
        rr.log(f"scene/{camera_name}",
               rr.Pinhole(
                       resolution=calibration['image_size'],
                       image_from_camera=K[:3, :3].flatten()
               ))
        image_path = os.path.join(image_folder, camera_name, os.listdir(os.path.join(image_folder, camera_name))[0])
        image = cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)
        rr.log(f"scene/{camera_name}", rr.Image(image))
        translation = np.array(calibration['translation'])
        R_inv = np.linalg.inv(calibration['rotation_matrix'])
        center = np.matmul(R_inv, -translation)
        rr.log(f"scene/{camera_name}", rr.Transform3D(mat3x3=R_inv, translation=center.flatten()))

Thanks for your help.

[abey79]

Thanks for the feedback. I'm trying to create a minimally reproducible example but have failed so far. All pinholes are properly displayed.

This is what I'm using (with rerun 0.10.1):

import math

import numpy as np
import rerun as rr

rr.init("rerun_example_pinhole_test")
rr.connect()
rng = np.random.default_rng(12345)

for i in range(16):
    image = rng.uniform(0, 255, size=[3, 3, 3])
    intrinsic = np.array([[4, 0, 1.5], [0, 4, 1.5], [0, 0, 1]])
    rr.log(f"scene/{i}", rr.Pinhole(resolution=[3, 3], image_from_camera=intrinsic))
    rr.log(f"scene/{i}", rr.Image(image))
    angle = i / 16 * np.pi * 2
    rr.log(f"scene/{i}", rr.Transform3D(
        mat3x3=[math.cos(angle), 0, math.sin(angle), 0, 1, 0, -math.sin(angle), 0, math.cos(angle)],
        translation=[i, 0, 0]))

Any chance you could build a minimal example that reproduces the issue?

Edit: maybe you could dump your calibrations structure to JSON. With it and your image files, we should be able to reproduce your exact setup.

[abey79]

Edit: nvm, I used a messed up rotation matrix.

What I did observe though is that the pinhole gizmo fails to render when that mat3x3 line is uncommented:

Maybe @Wumpf has an idea about what's going on here?

[Wumpf]

mat3x3=[math.cos(angle), 0, math.sin(angle), -math.sin(angle), 0, math.cos(angle), 0, 1, 0],

I might be wrong but that matrix looks a bit odd to me. Did you mean

mat3x3=[math.cos(angle), 0, math.sin(angle), 0, 1, 0, -math.sin(angle), 0, math.cos(angle)],

[gokcergun]

@abey79 Thank you for looking into the issue. I explored it a bit more. I can reproduce the error only when a subset of the images has a reverse resolution (portrait). You can find the required files to reproduce the error. The visualizer can sometimes display all 16 cameras together, mostly after the first run of the script. If I reset the viewer from the GUI, it displays either the set of images with normal resolution or the other set of images with reversed resolution.

Please let me know when you download the files.

[abey79]

I have downloaded the files and could successfully run the script 👍🏻 I'll report back ASAP.

[gokcergun]

Great, thank you