Skip to content
/ geod Public

[NeurIPS 2022 Spotlight] Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator

Notifications You must be signed in to change notification settings

VivianSZF/geod

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 

Repository files navigation

Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator

Figure: (a) Existing 3D-aware GANs where only the generator is made 3D-aware with the help of NeRF. (b) GeoD where the discriminator supervises the generator with the extracted geometry.

Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, Dit-Yan Yeung
Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)

[Paper] [Project Page] [Demo]

This work aims at improving 3D-aware image synthesis from the discriminator's perspective. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency.

Results

Qualitative comparison with pi-GAN as the base model.

3D reconstruction and novel view synthesis on real data with GAN inversion.

BibTeX

@inproceedings{shi2022improving,
    title   = {Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator},
    author  = {Shi, Zifan and Xu, Yinghao and Shen, Yujun and Zhao, Deli and Chen, Qifeng and Yeung, Dit-Yan},
    booktitle = {NeurIPS},
    year    = {2022}
  }

About

[NeurIPS 2022 Spotlight] Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published