This repository contains the PyTorch implementation of our paper, ReFiNe.
![](/TRI-ML/refine/raw/main/assets/tri-logo.png)
Sergey Zakharov
·
Katherine Liu
·
Adrien Gaidon
·
Rares Ambrus
SIGGRAPH, 2024
To set up the environment using docker, execute the following command:
make docker-build
Once installed, start an interactive session:
make docker-interactive
Once inside the interactive session, you can train and validate the method within this environment.
To replicate the workflow of our pipeline, first download 3 objects from the HB dataset.
Once downloaded, put them under the demo/
folder and unzip.
To generate GT files, run the following command:
python -m data.db_generate --config configs/config.yaml
The config.yaml
file stores default parameters for training and evaluation and points to the provided 3 models.
To start training, run the following script:
python train.py --config configs/config.yaml
To visualize the trained model using Open3D, run:
python visualize.py --path_net log/demo
This command will extract a dense point cloud from each of the decoded neural fields and visualize them sequentially. Press 9
to visualize normals, 1
to visualize RGB, -
and +
to decrease or increase the size of the points, and q
to proceed to the next object.
Additionally, you can specify the lod_inc
parameter to apply an increment on top of the default Level of Detail (LoD)
to further densify the output point cloud. By default, this parameter is set to 1.
python visualize.py --config configs/config.yaml --lod_inc 1
We provide pre-trained models on various datasets:
Dataset (GB) | # Objects | Latent | Size (MB) | Link |
---|---|---|---|---|
Thingi32 (0.47) | 32 | 64 | 3.4 | model |
ShapeNet150 (0.63) | 150 | 96 | 4.1 | model |
HB (0.53) | 33 | 64 | 4 | model |
BOP (0.91) | 201 | 512 | 92.8 | model |
GSO (13.6) | 1024 | 512 | 94.5 | model |
To visualize the pre-trained model, download it under the pretrained
folder and run:
python visualize.py --config pretrained/[model]/config.yaml
We used functions from NVIDIA's Kaolin and Kaolin Wisp libraries, as well as from Open3D, in our implementation.
@inproceedings{refine,
title={ReFiNe: Recursive Field Networks for Cross-Modal Multi-Scene Representation},
author={Sergey Zakharov, Katherine Liu, Adrien Gaidon, Rares Ambrus},
journal={SIGGRAPH},
year={2024}
}
This work is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International License.