Add pointcloud loading for blender parser

[gchhablani]

This PR adds pointcloud loading for blender parser.

The reason this is needed is because the original gaussian splatting repo supports blender format along with colmap.

With random initialization,splatfacto performance is not great (see below for lego).

Here are the relevant lines of code where they initialize the gaussians from pointcloud if that helps.

Note:

• I am not sure if transform_matrix and scaling_factor are needed here. If yes, any pointers on how it should be done will help.

[brentyi]

Hi @gchhablani, thanks for the PR! Where are you getting the point cloud from? The point clouds are usually pulled from COLMAP, but we don't typically run COLMAP for synthetic data since the training images are just rendered from known poses.

For what it's worth, my splatfacto results with the default settings are also not great:
ns-train splatfacto --data ./data/blender/lego blender-data

But I can get much cleaner results by setting the background color of the model to white, which is a typical hack that NeRF methods evaluated on synthetic data use:
ns-train splatfacto --data ./data/blender/lego --pipeline.model.background-color white blender-data

[brentyi]

#2889 also seems related!

[gchhablani]

@brentyi
I agree, there might be ways to fix things for this particular dataset. For synthetic unbounded datasets, partial pointclouds might be useful. Would it not be great to directly load ply files for pointcloud initializations for other blender format datasets?

I have ply files for sparse/imperfect pointclouds for a scene which are in blender format.

[gchhablani]

Btw, I get the pointcloud from the original nerf dataset.

[brentyi]

Btw, I get the pointcloud from the original nerf dataset.

Did you generate these yourself? I wasn't aware of any point clouds released in the original dataset, and can't seem to find them either (eg in this Google Drive).

[gchhablani]

@brentyi Thanks for asking me this :)
I wouldn't have checked otherwise.

I downloaded the synthetic data using this:

wget http://cseweb.ucsd.edu/~viscomp/projects/LF/papers/ECCV20/nerf/nerf_example_data.zip

Not sure where I had found this.

However, I checked the pointcloud since you asked.
And this is the pointcloud:

Interestingly, I get slightly better performance with this random cube, but this could be random too, of course:

[brentyi]

Assuming that's meant to be an RGB image: yeah, looks weird to me. It's interesting that the colors in the rendered eval image on the right look fine.

I'm not familiar with the tensorboard or wandb logging systems though.

[ichsan2895]

Hello, seems good PR. Please let me know some my thought

If I use my own blender model, where can I get sparse_pc.ply?

If I manually use ns-process-data images (which utilizes colmap) to get sparse_pc.ply, of course I will use nerfstudio dataparser instead of blender dataparser

[gchhablani]

@ichsan2895 @brentyi

The idea is to have it consistent with the original repo. I have some synthetic data for which I do have pointcloud in .ply but random init will not work on it.

Here is my use case:

• I have a set of images, corresponding intrinsic and extrinsics, and a pointcloud with holes in them.
• I converted them to Blender format with scripts to run with gaussian-splatting repository because that was the easiest thing.
• Now, to run things with nerfstudio dataparser, I will have to understand a new format and repeat the process.

Would it not be easier to just have pointcloud support in blender dataparser?

Regardless, if this PR doesn't make sense, I'll close it.
However, I would greatly appreciate some pointers on how to integrate transform_matrix and scaling_factor so I can use it myself, at least. Thanks!

[f-dy]

@brentyi @gchhablani to get reasonable result with the blender dataset, you should set random_scale (here https://github.com/nerfstudio-project/nerfstudio/blame/main/nerfstudio/models/splatfacto.py#L144 ) to the right value. It is 2.6, not 10 in 3DGS, see here https://github.com/graphdeco-inria/gaussian-splatting/blob/main/scene/dataset_readers.py#L240

That's something we noticed with @pa-la when working on #2889

I don't understand why it's 10 by default in nerfstudio? maybe for mipnerf360 scenes? I think we should set it to 2.6 by default, to be consistent with the original code.

[brentyi]

Sorry for the delay in getting back here! I've been hesitant here because it feels like the aim here would be to support non-standard dataset formats (AFAIK there's no public datasets in the Blender format that also provide point clouds).

But given how the uninvasive the changes are I'm fine with supporting it.

I think transform_matrix can be skipped (these are the camera poses? unless you're referring to something else), but the point should be scaled by scale_factor.

[brentyi]

can we replace these two fields with just one:

ply_path: Optional[Path] = None

?

and indicate in the docstring that this is used for Gaussian splatting initialization?

[brentyi]

I don't understand why it's 10 by default in nerfstudio? maybe for mipnerf360 scenes? I think we should set it to 2.6 by default, to be consistent with the original code.

Maybe @kerrj can comment on this?

[kerrj]

I don't understand why it's 10 by default in nerfstudio? maybe for mipnerf360 scenes? I think we should set it to 2.6 by default, to be consistent with the original code.

We can change back to 2.6, the intention with setting to 10 was that if people use real datasets without points available (eg Polycam), initializing to 10 would give reasonable results on backgrounds sometimes, while 2.6 would look bad. I'm fine changing the default to 2.6, since point pipelines for other data inputs should be added now anyways.

[gchhablani]

@brentyi I have updated the PR. Requesting for another review.

[jb-ye]

@brentyi @kerrj Any follow up on this? It looks like a simple non-invasive change.

[brentyi]

Thanks for the bump; I fixed the scale factor and made the option opt-in. Should be okay with me now!