Cannot process data generated by Blender (mp4)

[XinyueZ]

Is it actually possible to use the output from Blender?

Those are the feedback:

Number of frames in video: 500
( ●    ) Converting video to images...Can't satify requested number of frames. Extracting all frames.
[13:03:22] 🎉 Done converting video to images.                                                 process_data_utils.py:107
[13:03:31] 🎉 Done downscaling images.                                                         process_data_utils.py:214
[13:03:38] 🎉 Done extracting COLMAP features.                                                       colmap_utils.py:530
[13:04:11] 🎉 Done matching COLMAP features.                                                         colmap_utils.py:544
◠ Running COLMAP bundle adjustment... (This may take a while)──────────────────────────────────────────────  💀 💀 💀 ERROR 💀 💀 💀  ───────────────────────────────────────────────
Error running command: colmap mapper --database_path data/cust_data_3_processed/colmap/database.db --image_path 
data/cust_data_3_processed/images --output_path data/cust_data_3_processed/colmap/sparse 
--Mapper.ba_global_function_tolerance 1e-6

[tancik]

It looks like COLMAP failed to register the frames. If you are working with synthetic data, I'd recommend exporting the ground truth camera frames instead of using COLMAP.

[Mason-McGough]

Are the instructions for exporting camera transforms and frames from custom Blender scenes included in this repo? If not, I can add them in a PR.

[XinyueZ]

@Mason-McGough pls

[Mason-McGough]

A commit was just pushed recently related to issue 1218. Seems like it only applies to camera paths for rendering and not for producing training data, if I'm not mistaken. I have seen projects like BlenderNeRF for this sort of thing, but does nerfstudio have a native workflow for importing Blender images and poses as training data?

[XinyueZ]

A commit was just pushed recently related to issue 1218. Seems like it only applies to camera paths for rendering and not for producing training data, if I'm not mistaken. I have seen projects like BlenderNeRF for this sort of thing, but does nerfstudio have a native workflow for importing Blender images and poses as training data?

I have just imported an mp4 created by Blender, the studio, I guess that the COLMAP complained about it.
Maybe the studio could contain some other projects like COLMAP, just my opinion.

[XinyueZ]

I can provide the mp4, you can try it self @Mason-McGough

NeRF.Test.Scene.mp4

[Mason-McGough]

Just a guess, but judging by the video your problem could be due to a lack of image features. SfM-based pose solvers like COLMAP need detailed geometry in order to find feature correspondences. This video mostly consists of objects with smooth featureless textures. Could you apply the same workflow to a real-world video or to a synthetic scene with more detail to see if the poses can be solved? In the meantime, I will look into the best way to export camera frames and poses from Blender.

[Mason-McGough]

@XinyueZ I was able to replicate your issue with that video in COLMAP. I will continue to look into that. For now, I was able to successfully train a Nerfacto model with ground-truth poses using the BlenderNeRF project and a free scene I downloaded from TurboSquid. All I had to do was rename "transforms_train.json" to "transforms.json" to get it to work.

Here is an example render I made with it. You might be able to use BlenderNeRF for your needs for now.

cozy-room_train_2023-01-28_092527.mp4

[iraj465]

Hi @tancik this is also failing for me, sometimes the mapper step goes through and sometimes it fails for the same video. BlenderNeRF i found takes a bit more time than the nerfstudio process data utils. Any reason why this is failing?