HAHA: Highly Articulated Gaussian Human Avatars with Textured Mesh Prior

Project page: https://david-svitov.github.io/HAHA_project_page/
Arxiv paper: https://arxiv.org/pdf/2404.01053.pdf
YouTube video: https://youtu.be/vBzdAOKi1w0

We present HAHA - a novel approach for animatable human avatar generation from monocular input videos. The proposed method relies on learning the trade-off between the use of Gaussian splatting and a textured mesh for efficient and high fidelity rendering. We demonstrate its efficiency to animate and render full-body human avatars controlled via the SMPL-X parametric model. Our model learns to apply Gaussian splatting only in areas of the SMPL-X mesh where it is necessary, like hair and out-of-mesh clothing. This results in a minimal number of Gaussians being used to represent the full avatar, and reduced rendering artifacts. This allows us to handle the animation of small body parts such as fingers that are traditionally disregarded. We demonstrate the effectiveness of our approach on two open datasets: SnapshotPeople and X-Humans. Our method demonstrates on par reconstruction quality to the state-of-the-art on SnapshotPeople, while using less than a third of Gaussians. HAHA outperforms previous state-of-the-art on novel poses from X-Humans both quantitatively and qualitatively.

Installation

Clone the repository

git clone --recursive https://github.com/david-svitov/HAHA.git

Environment

The easiest way to build an environment for this repository is to use docker image. To build it, make the following steps:

1. Build the image with the following command:

bash docker/build.sh

2. Start a container:

bash docker/run.sh

It mounts root directory of the host system to /mounted/ inside docker and sets cloned repository path as a starting directory.

3. Inside the container install diff_gaussian_rasterization-alphadep. (Unfortunately, docker fails to install it during image building)

bash gaussian_install.sh

Download dependencies

SMPL-X

Download SMPL-X v1.1 models_smplx_v1_1.zip from SMPL-X official website and extract files to get:

models/smplx
    ├── SMPLX_FEMALE.pkl
    ├── SMPLX_FEMALE.npz
    ├── SMPLX_MALE.pkl
    ├── SMPLX_MALE.npz
    ├── SMPLX_NEUTRAL.pkl
    ├── SMPLX_NEUTRAL.npz

You will need to change the path to the models' folder in the config parameter smplx_path of the config in ./configs/*.yaml you are going to use.

AMASS for novel poses rendering

Download the SMPL-X N package of CMU subset from AMASS and unzip it.

You will need to change the AMASS dataset pathin the config parameter animation_sequence of the config in ./configs/*.yaml you are going to use.

Evaluation

Download pretrained models:

PeopleSnapshot

white - Results rendered with white background
black - Results rendered with black background

Male-3-casual	Female-3-casual	Male-4-casual	Female-4-casual
Checkpoint	Checkpoint	Checkpoint	Checkpoint
Renders (white)	Renders (white)	Renders (white)	Renders (white)
Renders (black)	Renders (black)	Renders (black)	Renders (black)
Renders (Novel poses)	Renders (Novel poses)	Renders (Novel poses)	Renders (Novel poses)

Or Download all checkpoints and renders.

X-Humans

For X-Humans all renders are provided in novel poses.

00016	00019	00018	00027
Checkpoint	Checkpoint	Checkpoint	Checkpoint
Renders (white)	Renders (white)	Renders (white)	Renders (white)
Renders (black)	Renders (black)	Renders (black)	Renders (black)

Or Download all checkpoints and renders.

Table with names of the sequences used in the experiments

	00016	00019	00018	00027
Train sequence path	/train/Take4	/train/Take5	/train/Take6	/train/Take7
Test sequence path	/train/Take6	/train/Take7	/train/Take9	/train/Take6

Render

To generate an animation with an avatar, use the following command:

python main.py --base=./configs/<config name>.yaml --pretrained=/<checkpoints>/OPTIMIZE_OPACITY_10500.ckpt --test_mode

Training

Datasets preprocessing

SnapshotPeople

We use the data from InstantAvatar and finetuned their pre-processed poses to get SMPL-X fits.

Preprocessed data with SMPL-X fits can be downloaded here.

X-Humans

The data can be requested and downloaded here.

No special preprocessing is required.

Training

To train an avatar, run the code as follows. Make sure to modify all necessary paths in the config file.

python main.py --base=./configs/<config name>.yaml

Citation

If you find this code useful for your research, please consider citing:

@misc{svitov2024haha,
      title={HAHA: Highly Articulated Gaussian Human Avatars with Textured Mesh Prior}, 
      author={David Svitov and Pietro Morerio and Lourdes Agapito and Alessio Del Bue},
      year={2024},
      eprint={2404.01053},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}