Adding glomap support to ns-process-data, works with COLMAP 3.8!

[AntonioMacaronio]

Problems and Background

• GLOMAP is orders of magnitudes faster than COLMAP. On average, it finds camera extrinsics, intrinsics, and other distortion parameters and such 100x faster than COLMAP, and it would great to support this in Nerfstudio and already has been requested

Overview of Changes

• Adding a GLOMAP toggle in ns-process-data script, which uses the GLOMAP mapper
• Updated the loading messages

[kerrj]

This is great! It would also be good to write documentation or link to installation instructions and the like

[hyqshr]

Any update on this?

[jb-ye]

@AntonioMacaronio Do you want to rebase your changes on top main? I think this one is close to merge.

[AntonioMacaronio]

@jb-ye Sure! I'll need to do one last round of testing and I'll fix the build errors and this should be good to go

[AntonioMacaronio]

@jb-ye update on this: there is currently a bug with how the cameras are placed in the viewer and during training. In the colmap visualizer, the camera poses are fine and look accurate. However, during training in the nerfstudio viewer all the cameras get placed in 1 spot at the origin. I'll have to look into the nerfstudio_dataparser.py to see the issue with this

[pablovela5620]

@jb-ye update on this: there is currently a bug with how the cameras are placed in the viewer and during training. In the colmap visualizer, the camera poses are fine and look accurate. However, during training in the nerfstudio viewer all the cameras get placed in 1 spot at the origin. I'll have to look into the nerfstudio_dataparser.py to see the issue with this

@AntonioMacaronio
I think (?) this is may be due to lack of normalization. What commit of glomap are you using? They recently add this fix in this commit
colmap/glomap@d41b747
I had a similar problem but it got fixed this this update
colmap/glomap#128 (comment)

[AntonioMacaronio]

@pablovela5620 I just updated my glomap to master's latest commit, rebuilt, and tried to run ns-process-data, but the cameras still get placed at the origin when using the nerfstudio viewer; have you had any success with this PR? Sometimes the nerfstudio viewer doesn't place everything at the origin and the poses look accurate but because it's inconsistent I'm hesitant on merging

[pablovela5620]

It turns out there's another issue, and you're right to be hesitant! I've been doing some debugging, and as you stated, even with the latest PR, you get the weird clustering effect. I basically ran colmap/glomap (not with this pr but with my own implementation using a different feature+matcher combo). I found that glomap can have some catastrophic failures for a small number of cameras out of the whole group. I linked a video showing this, where I'm comparing a reconstruction of 250 images using the aliked feature detector and light glue matcher with colmap and then with glomap.

Basically one of the cameras is WAY out of bounds leading to the clustering at the origin, one or two cameras end up being 10x farther out due to a failure in point triangulation(?). I'm going to open an issue on the glomap repo

glomap-failure-featureless.webm

[AntonioMacaronio]

Basically one of the cameras is WAY out of bounds leading to the clustering at the origin, one or two cameras end up being 10x farther out due to a failure in point triangulation(?).

I saw your comment on the glomap issues page with the video of the COLMAP gui and I see the exact same thing with my COLMAP gui. Most of the cameras are close together and accurate but one of them is very far away. What's puzzling is that Nerfstudio compacts all the close cameras to the origin. Even if one of the cameras is far away, because Nerfstudio is scaling the distances between the cameras, it should still remain proportional (in my thinking), so I'm still not sure why Nerfstudio crams all the cameras at the same spot - there should still be some space between the cameras

[pablovela5620]

Yea, I'm not sure why it fails in Nerfstudios case. Might be worth looking into the normalization code that Glomap added to see how they fixed it. In either case, I think until these outlier cases get resolved its probably not good to merge this is. The massive outliers completely ruin the reconstruction into something nonsensical. Maybe there could be some heuristic value that tries to deal with outliers (like projection error or if the camera is 10x further from the center of mass or something) but those still feel very hacky