Volumetric sampling bug/problems in Instant-NGP implemtation

[dozeri83]

Hi @tancik ,

---

tl;dr - there is a problem in the volumetric sampler resulting in a memory crash.
The cause of this issue is the initial sigma_fn method in the ray marching.

---

I have been experimenting for a while in different configurations and methods in nerfstudio and stumbled upon weird issue.
First of all I am using instant-ngp (with AABB contraction) implementation since I can compare it visually to the original instant-ngp.

I am using blender hotdog data set with aabb 1. (on nerfstudio - scene_scale 0.5)

nerfstudio seem to crash on GPU memory because of this line

nerfstudio/nerfstudio/pipelines/dynamic_batch.py

Line 74 in 6baf31d

self.dynamic_num_rays_per_batch * (self.config.target_num_samples / num_samples_per_batch)

num_samples_per_batch - was very small.

the reason num_samples_per_batch was very small is because on the first iteration of the train the volumetric sampler
rejected the samples because of sigma_fn ( the samples had zero density so it rejected all of them)

nerfstudio/nerfstudio/model_components/ray_samplers.py

Line 458 in e605b33

packed_info, starts, ends = nerfacc.ray_marching(

I had 2 workarounds -

1. replaced sigma_fn with None.
2. replaced sigma_fn with None for the first 100 iteration and then replaced it with self.get_sigma_fn(rays_o, rays_d).

workaround 1 work best for me (uniform sampler?) with better PSNR.

BTW - nerfacto model looked very bad on the blender hotdog data set. ( but this is a different issue)

[dozeri83]

Reconstruction:

ns-train instant-ngp --trainer.max-num-iterations 35000 --vis tensorboard --experiment-name transforms_train_scaled_0_3333333 --output_dir output/hotdog --timestamp 23 --data nerf_synthetic/hotdog/ --orientation-method none --center-poses False --auto-scale-poses False --scene_scale 0.5

where I scaled the transforms file by 0.3333 (ngp scale) and used nerstudio loader and change the contraction type to ContractionType.AABB:

transforms.zip

[tancik]

I think the crux of the issue is that instant-ngp and nerfacto have been optimized to work with real-world data, not the blender data. (ie you can see a discussion in #806). The best solution is probably to have multiple models specific to different usecases, ie a "blender" version of instant-ngp and nerfacto (it would use a different sampling strategy, no pose opt, no appearance opt, no contraction...). On our end the question is how to prioritize improving real-world datasets for applications vs datasets like blender for research benchmarks.

[dozeri83]

I would like to split this discussion into two parts:

1. a. would you consider changing instant nerf default contraction method to AABB method ? ( asking because this is the contraction method of the original instant-nerf)
b. I would also like to add a small protection code to the volumetric sampler - say, if the number of samples is below some threshold, ignore the density function.
I can implement those changes to a pull request.

2. I agree that if the scene is unbounded then instant ngp will preform poorly. However if the scene is actually bounded then there are certain real-world scenarios where instant ngp method looks better than nerfacto.

For example - arial mapping drones :
I used this dataset (dronemapper)
https://dronemapper.com/drone-mapping-cedaredge-4th-avenue-reservoir-capacity-estimate/

nerfacto :

This scene is very fogy - even on "untrained areas"

instant-ngp: (after the fixes I mentioned)
untrained areas have density zero.

also - it has better test PSNR.

It would be great if nerfacto can match this but I think that the AABB contraction has advantages on bounded real-world scenes.

[tancik]

I think the way to go about this is to create a new config, "instant-ngp-bounded" that contains these changes so that people can choose depending on the data they are working with.