Regarding Rectification Map

[Ethan-Zhou]

Hi,

I found the dispaity_events GT were misaligned with the rectified event map (obtained by accumulating a number of events on the image plane without motion compensation). To double check this, I would like to confirm the follwing definition with you.

I wonder if the provided rectify_map is equivalent to that used by the cv.remap function (https://docs.opencv.org/3.4/da/d54/group__imgproc__transform.html#gab75ef31ce5cdfb5c44b6da5f3b908ea4).

The "rectify_map" looks like the forward mapping funciton, while the one used by the cv.remap function is the inverse mapping function, right?

Thanks in advance.

[magehrig]

Hi @Ethan-Zhou

The rectify_map.h5 file contains the forward mapping from the distorted (original) pixel coordinates to the rectified pixel coordinates of the event cameras. As you mentioned, cv.remap does backward warping instead.

Can you specify, how you came to the conclusion that the GT is not aligned properly? Do you have a code example that I can look at and a visualization? What exactly is the procedure and on which sequence(s)?

[Ethan-Zhou]

Hi, Mathias
Here is what I did:

1. I created a time surface (tsMap_xx) with raw events, and tried to undistort+rectify them (tsMap_xx_rect) withthe following code.

tsMap_left_rect = cv2.remap(tsMap_left.numpy().astype("uint8"), rectifyMap_left[:,:,0], rectifyMap_left[:,:,1], cv2.INTER_LINEAR)#, cv2.BORDER_ISOLATED)
tsMap_right_rect = cv2.remap(tsMap_right.numpy().astype("uint8"), rectifyMap_right[:,:,0], rectifyMap_right[:,:,1], cv2.INTER_LINEAR)#, cv2.BORDER_ISOLATED)

2. Then I aligned the disparity_events GT with this undistorted + rectified time surface map (let's use tsMap_left_rect as an example) with the example function "get_disp_overlay_rgb" provided by you. Here is what I got

An apparent misalignment is witnessed.

I also tried to overlap disparity_image GT to the corresponding rgb frame. Here is what I got

This alignment looks good to me.

I have been only testing the sequnence "interlaken_00_c" so far.

It seems that I should compute the inverse mapping by my self and use it in the remap function then, right?

Cheers,
yi

[magehrig]

You cannot use cv2.remap with the h5 rectification maps that are provided. Let me explain in more detail.

There are two ways to achieve the rectification of event data.

1. Forward warping of event pixel coordinates: This requires a mapping of distorted pixel coordinates to rectified pixel coordinates, which is what I provide with the rectify_map.h5 files. That is, for each raw event (with integer pixel coordinates) I can retrieve the coordinates it will have in the rectified frame, resulting in floating-point coordinates for event data. This approach is implemented in the code that I provide; the rectification map was computed with cv2.undistortPointsIter. Note that this is different from how images are typically rectified, which I mention next.
2. Backward warping of an event representation: This is what we usually do for rectifying images, as you probably know very well. This requires a mapping from rectified pixel coordinates to distorted pixel coordinates, which is the opposite direction of the previous approach. Such a mapping can be computed using cv2.initUndistortRectifyMap. Subsequently, cv2.remap can be directly used to remap a distorted image to its rectified version. Similarly, this can be done with an image-like representation such as time surfaces (or even voxel grids). This is the approach you chose with the previous post.

What went wrong? You were mixing 1) and 2) by backward warping with the forward warping mapping.

How can you resolve this issue? Either use 1) or fix 2). Approach 1) is already implemented in the dataloader that I provide so I am briefly explaining how fix 2) for the example of the left event camera (cam index 0):

# Pseudo code for loading calibration from cam_to_cam.yaml per sequence:
# K_dist <- intrinsics[cam0][camera_matrix]
# dist_coeffs <- intrinsics[cam0][distortion_coeffs]
# R_rect0 <- extrinsics[R_rect0]
# K_rect <- intrinsics[camRect0][camera_matrix]
# resolution <- intrinsics[camRect0][resolution]
#
# representation <- load your favorite event representation

mapping = cv2.initUndistortRectifyMap(K_dist, dist_coeffs, R_rect0, K_rect, resolution, cv2.CV_32FC2)[0]
rect_representation = cv2.remap(representation, mapping, None, interpolation=cv2.INTER_CUBIC)

Note that approach 2) cannot be used to rectify raw events directly because the problem is ill-posed (except if nearest-neighbor "interpolation" is used). Backward warping attempts to reconstruct a pixel value via interpolation of neighboring pixels. However, for event data the value of neighboring pixels is only defined exactly when an event appears such that a naive application of backward warping fails. This is why 2) only applies to image-like representations of event data.

Let me know if this resolves this issue.

[magehrig]

Hi, Mathias
Here is what I did:

1. I created a time surface (tsMap_xx) with raw events, and tried to undistort+rectify them (tsMap_xx_rect) withthe following code.

tsMap_left_rect = cv2.remap(tsMap_left.numpy().astype("uint8"), rectifyMap_left[:,:,0], rectifyMap_left[:,:,1], cv2.INTER_LINEAR)#, cv2.BORDER_ISOLATED)
tsMap_right_rect = cv2.remap(tsMap_right.numpy().astype("uint8"), rectifyMap_right[:,:,0], rectifyMap_right[:,:,1], cv2.INTER_LINEAR)#, cv2.BORDER_ISOLATED)

1. Then I aligned the disparity_events GT with this undistorted + rectified time surface map (let's use tsMap_left_rect as an example) with the example function "get_disp_overlay_rgb" provided by you. Here is what I got

An apparent misalignment is witnessed.

I also tried to overlap disparity_image GT to the corresponding rgb frame. Here is what I got

This alignment looks good to me.

I have been only testing the sequnence "interlaken_00_c" so far.

It seems that I should compute the inverse mapping by my self and use it in the remap function then, right?

Cheers,
yi

TLDR: yes, I compiled a long answer if anyone else ever has a similar question and finds this issue :)

[Ethan-Zhou]

Hi Mathias,

Thanks so much for the detailed guidance. It perfectly solves my problem. Here attached is the new result of the alignment.