Question about σ-fusion

[Willyzw]

Hi, thanks for your great work, which also includes the implementation of sigma-fusion.

In the NeRF-SLAM paper, Table 1 shows a quite large difference in depth L1 metric between the TSDF-based methods (besides σ-fusion) and the nerf-based ones, which has surprised me a lot. So I just run the experiments and test this code myself. The Depth L1 results are as follows:

it turns out the TSDF-based approaches are indeed not as good as nerf-slam but actually not that much worse. I'm curious if this is related to any updates or bug fixes that have been made since then.

[phamquandung]

Hi @Willyzw. How can you get that result when just running the code: python ./examples/slam_demo.py --dataset_dir=./datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='sigma' --gui

Could you please help me? Thanks in advance

[Willyzw]

Hi @phamquandung, first you need to get it installed, I mostly get help from this PR #12. To run the evaluation, you will also need to add the flag --eval to the command.

In addition, I've committed my changes here, which may be useful afc1c2f

[ToniRV]

Thanks for running the eval again @Willyzw!!

I might have made a mistake.
Your numbers make more sense than mine I think, but it is difficult to say.
Did you change something in the evaluation code?
Could you at least reproduce the results for nerf-slam?

I can't re-run the eval right now, but, if you could share all the results you got, we can make sure we are getting similar values.

[Willyzw]

Hi @ToniRV glad to receive your reply!

You can please check my changes here master...Willyzw:NeRF-SLAM:master
The results were run after the first commit, so I hardly change any functional stuff or parameter. Later I increase the block_size and mask out the pixels that lack gt values, which has minor improvements. Follows are my updated results:

Besides notably the PSNR values are exactly the same as on paper.

Yes, I also run the nerf fusion part. The results are almost consistent with the paper. An exception is the sequence office-1, somehow my L1 is 10.28 vs. 1.98 on paper. I guess the sequence is quite challenging due to the darkness of the scene. The results of nerf-slam are as follows:

Above are all my results. Cheers!

[phamquandung]

Hi @Willyzw. Have you tried with NeRF? python ./examples/slam_demo.py --dataset_dir=./datasets/Replica/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --eval. My result is all zero. I tried with with sigma but then it just stopped.

[Willyzw]

Hi @phamquandung, yes I have tried both and have also encountered the mentioned problems. For nerf fusion, you need to skip the fitting step in the very beginning when missing images by checking n_images_for_training, otherwise it just finishes all training steps with 0 images. Or you can try to run the sequential mode.

For sigma fusion, I remember you need to disable the dataflow coming from the dataloader in slam_demo.py.

[phamquandung]

Hi @Willyzw. Thanks for you quick response. You mean I need to check the line 74 in nerf_fusion.py as below figure? But then I meet other problem as shown below.

[xiehousen]

Hi @phamquandung, yes I have tried both and have also encountered the mentioned problems. For nerf fusion, you need to skip the fitting step in the very beginning when missing images by checking n_images_for_training, otherwise it just finishes all training steps with 0 images. Or you can try to run the sequential mode.

For sigma fusion, I remember you need to disable the dataflow coming from the dataloader in slam_demo.py.

Hi @Willyzw ,
I wonder how to skip the fitting step?
In my test, the self.ref_frames and the n_images_for_training are both zero.

Thanks a lot for your reply.

[Willyzw]

Hi @phamquandung I didn't encounter your problem. Maybe you better try the sequential mode.
@xiehousen where do you test the variable? If it is in the constructor, it is no wonder all zeros.

[eriksandstroem]

Hi @phamquandung, yes I have tried both and have also encountered the mentioned problems. For nerf fusion, you need to skip the fitting step in the very beginning when missing images by checking n_images_for_training, otherwise it just finishes all training steps with 0 images. Or you can try to run the sequential mode.
For sigma fusion, I remember you need to disable the dataflow coming from the dataloader in slam_demo.py.

Hi @Willyzw , I wonder how to skip the fitting step? In my test, the self.ref_frames and the n_images_for_training are both zero.

Thanks a lot for your reply.

I am also facing this issue. Would appreciate if you have any input @Willyzw. The variable self.npg.nerf.training.n_images_fro_training is updated, but the ref_frames list is empty.

Edit: After looking at your changes to the code here master...Willyzw:NeRF-SLAM:master and adding the two lines at row 287 where it says "# TODO: we need to restore the self.ref_frames[frame_id] = [image, gt, etc] for evaluation...." in nerf_fusion.py it now runs. Strange that this is not updated in the official code release yet.