DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation

Jiwook Kim^*, Seonho Lee^*, Jaeyo Shin, Jiho Choi, Hyunjung Shim

(* indicates equal contribution)

This is the official implementation of DreamCatalyst.

DreamCatalyst.yuseung.mov

News & Updates

• [TBA] ✨ Codes based on the Threestudio for our method with Gaussian Editor will be released soon.

• [2024/10/02] 🌟 Codes based on the Nerfstudio have been released!

• [2024/07/17] 📄 Our paper is now available! The paper is released here.

Introduction

DreamCatalyst is a novel framework that considers the sampling dynamics of diffusion models in the SDS(Score Distillation Sampling) framework. DreamCatalyst can be applied to various models in 3D editing, such as NeRF and 3DGS. This method aims to reduce training time and improve editing quality. Please refer to the paper for more details.

Architecture

Get Started

We provide the implementation of DreamCatalyst based on the both NerfStudio and Threestudio frameworks. Nerfstudio is a framework for NeRF or 3DGS editing. Threestudio is a framework for our method with Gaussian Editor. Please refer to the following instructions.

Dataset

Please download the dataset from the following link and extract it to the dataset directory.

PDS provides image data for 3D scene. Please download the dataset here. This dataset includes 3D scene data from Instruct-NeRF2NeRF as a subset. You do not need to download Dreambooth checkpoints since our method does not require them.

For NerfStudio

Please refer to nerfstudio/README.md.

For Threestudio

We will release it soon.

Citation

If you find our work useful in your research, please cite:

@misc{kim2024dreamcatalystfasthighquality3d,
      title={DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation}, 
      author={Jiwook Kim and Seonho Lee and Jaeyo Shin and Jiho Choi and Hyunjung Shim},
      year={2024},
      eprint={2407.11394},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.11394, 
}

Acknowledgement

We would like to express our gratitude to the following works:

• PDS
• Instruct-Nerf2Nerf
• InstructPix2Pix
• Gaussian Editor
• FreeU
• DreamSampler
• SDEdit