GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces

Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long*, Wenping Wang, Yuexin Ma*

*Corresponding authors

Paper | Project Page

Introduction

The advent of neural 3D Gaussians has recently brought about a revolution in the field of neural rendering, facilitating the generation of high-quality renderings at real-time speeds. However, the explicit and discrete representation encounters challenges when applied to scenes featuring reflective surfaces. In this paper, we present GaussianShader, a novel method that applies a simplified shading function on 3D Gaussians to enhance the neural rendering in scenes with reflective surfaces while preserving the training and rendering efficiency.

GaussianShader maintains real-time rendering speed and renders high-fidelity images for both general and reflective surfaces. GaussianShader enables free-viewpoint rendering objects under distinct lighting environments.

GaussianShader initiates with the neural 3D Gaussian spheres that integrate both conventional attributes and the newly introduced shading attributes to accurately capture view-dependent appearances. We incorporate a differentiable environment lighting map to simulate realistic lighting. The end-to-end training leads to a model that reconstructs both reflective and diffuse surfaces, achieving high material and lighting fidelity.

Installation

Provide installation instructions for your project. Include any dependencies and commands needed to set up the project.

# Clone the repository
git clone https://github.com/Asparagus15/GaussianShader.git
cd GaussianShader

# Install dependencies
conda env create --file environment.yml
conda activate gaussian_shader

Running

Download the example data and put it to the data folder. Execute the optimizer using the following command:

python train.py -s data/horse_blender --eval -m output/horse_blender -w --brdf_dim 0 --sh_degree -1 --lambda_predicted_normal 2e-1 --brdf_env 512 

Rendering

python render.py -m output/horse_blender --brdf_dim 0 --sh_degree -1 --brdf_mode envmap --brdf_env 512

Dataset

We mainly evaluate our method on NeRF Synthetic, Tanks&Temples, Shiny Blender and Glossy Synthetic. You can use nero2blender.py to convert the Glossy Synthetic data into Blender format.

More features

The repo is still being under construction, thanks for your patience.

• Arguments explanation.
• Residual color training code.

Acknowledgement

We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.

• gaussian splatting
• Ref-NeRF
• nvdiffrec
• Point-NeRF

Citation

If you find this repository useful in your project, please cite the following work. :)

@article{jiang2023gaussianshader,
  title={GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces},
  author={Jiang, Yingwenqi and Tu, Jiadong and Liu, Yuan and Gao, Xifeng and Long, Xiaoxiao and Wang, Wenping and Ma, Yuexin},
  journal={arXiv preprint arXiv:2311.17977},
  year={2023}
}