Add support for RealityCapture

[tancik]

Not fully working yet. Poses seem good for some datasets, but not all. Still debuging.

[machenmusik]

Some quick questions

• Looking at RealityCapture export options, there are many; which CSV export option(s) are supported? (Answer seems to be "Internal / external camera parameters")
• Would it help to use non-CSV alignment export options such as Bundler v0.3? (Probably not, assuming this works)

[machenmusik]

FYI, I found that this needed #1057 as well as same fix in another spot.

nerfstudio\data\dataparsers\nerfstudio_dataparser.py:215
idx_tensor = torch.tensor(indices).type(torch.long)

[machenmusik]

(Thanks for better suggestion #1057 (comment) )

[machenmusik]

(Any update?)

[tancik]

(Any update?)

Sorry for the lack of updates, other tasks pushed this down the priority stack. It currently mostly works, but the scene is currently flipped and the quality seems a bit blurrier than expected. Im not sure if this is a problem with how I am processing he poses, or if the data I am testing with has issues. I am also unsure if the camera model I am using for distortions exactly matches the one that realitycapture uses. I hope to get back to this PR at somepoint, but if probably won't be anytime in the immediate future. Others are welcome to hop onto this PR to see if they can fix / answers some of the issues I've listed above.

[machenmusik]

Did a quick test, something is definitely strange

• compared to Polycam, and COLMAP processing for same image set, RealityCapture values for focal length seem off
• using Polycam/COLMAP values with RealityCapture transforms, one gets much better quality, but it appears mirrored, so maybe LHS/RHS issue

[machenmusik]

With the output sample I tried, and the changes mentioned, I was able to get a quite good nerfacto result, with the only annoyance being that I needed to rotate the view 180 degrees.

Obviously we want more than one sample to try in order to confirm.
(Unfortunately I am not at liberty to share my output sample.)

[dkorolov]

I make some own debugging related RealityCaprure data import. We have some mess in nerfstudio related to the Orientation EXIF tag in source images and FFMPEG images autorotation behavior. This significantly influences on RealityCaprure data import.
I created a topic related to this
#1260

[dkorolov]

I tested the autorotation issue on my data I can not share. But for public testing, we can use the Gingerbread Man dataset (it is free for everyone) - https://www.capturingreality.com/download/files/Calendar-gingerbread-man
Also here are many free datasets that come with RealityCapture - https://www.capturingreality.com/free-datasets.

[machenmusik]

Thanks @dkorolov.
The original issues with this PR branch appear to be resolved;
wondering if we want to clean up EXIF orientation metadata behavior in a subsequent PR.
@tancik thoughts?

[machenmusik]

Some good news, looks like -noautorotate makes the Gingy sample process correctly. Incoming separate PR #1261.

[machenmusik]

When rebased against latest main including #1261, this PR now successfully processes data and trains against both CottageMiniature (ordinary images) and Gingy (rotated images) sample sets from RealityCapture, yielding nice looking results (at least to my eye).

[tancik]

@machenmusik Thanks for helping out with this!