Lightning Flash Integration

[tchaton]

Dear people from Open3d-ML,

My name is Thomas Chaton, I am part of the PyTorch Lightning Team and co-creator of TorchPoints3d: https://github.com/nicolas-chaulet/torch-points3d
This week, I will integrate Open3d-ML in Lightning Flash: https://github.com/PyTorchLightning/lightning-flash.

Any help such as reviews would be appreciated :)

Best,
T.C

[tchaton]

Dear people from Open3d-ML,

Actually, more examples for all datasets would be highly appreciated. The resources are pretty sparse :)

Observations (will be added there as I go through the codebase):

• Are preprocess and transform called within the TorchDataLoader ? Why TorchDataLoader for a dataset, this is confusing.
• It seems unclear how the model refers to a given task.
• It isn't clear which version of S3DIS you were using. The 1x1 dataset or the original one ?

Here is the PR: Lightning-Universe/lightning-flash#566

Best,
T.C

[tchaton]

Here is the current API.

And associated PR: https://github.com/PyTorchLightning/lightning-flash/pull/566/files

Best,
T.C

[sanskar107]

@tchaton We appreciate your efforts in integration of Open3D-ML in Lightning Flash, and we are happy to help.

We are working on adding more examples and tutorials. and to answer some of your quesitons :

1. Yes, preprocess and transform are called within the TorchDataLoader. The results of preprocess can be cached to save computation.
   "Why TorchDataLoader for a dataset ?". We support both PyTorch and Tensorflow. Out dataset classes are independent of any ML framework and we wrap them with TorchDataLoader and TFDataLoader to make it compatible with PyTorch and TensorFlow respectively.
2. Tasks are defined by pipeline. Feel free to ask a detailed question.
3. We use Stanford3dDataset_v1.2_Aligned_Version which is used in this paper http://buildingparser.stanford.edu/images/3D_Semantic_Parsing.pdf

[tchaton]

Dear @sanskar107,

Thanks for answering my questions ! I will work on integration ObjectDetection this week.

Here is the PointCloudSegmentation Task: https://lightning-flash.readthedocs.io/en/latest/reference/pointcloud_segmentation.html

Would be nice to provide more pre-trained models, I could help there and maybe bring some models from TorchPoints3d.

Note: I noticed the visualiser was quite un-stable and there were a bug in the base Visualizer class. I had to patch it: https://github.com/PyTorchLightning/lightning-flash/blob/master/flash/pointcloud/segmentation/open3d_ml/app.py#L28

Best,
T.C

[tchaton]

Dear @sanskar107 @YilingQiao

Would you be interested to have chat with the TorchPoints3d, Lightning Flash team.
We are thinking to centralise efforts behind 1 library and would like to potentially discuss a merger.

Best,
T.C