The FID result cannot be aligned with pytorch-fid/torch-fidelity

[SingleZombie]

🐛 Describe the bug

Just use FrechetInceptionDistance on any images.

import torch
from torcheval.metrics import FrechetInceptionDistance
imgs_1 = ...
imgs_2 = ...
fid = FrechetInceptionDistance(device=device)
fid.update(imgs_1 , True)
fid.update(imgs_2 , False)
print(fid.compute())

I have found the causes.

1. The model and weight of InceptionV3 is not strictly same to the TensorFlow version, which is used by most papers. Now I replace it with InceptionV3 model from pytorch-fid.
2. The insufficient precision of torcheval.metrics.image.fid.FrechetInceptionDistance._calculate_frechet_distance leads to wrong results. I fix this by setting all variables (fake_sum, real_cov_sum, ...) to float64.

With these modifications, the FID results are aligned. I suggest the maintainers reimplement the FID to align it with other libraries, otherwise researchers won't use torcheval to calculate FID.

Versions

PyTorch version: 2.0.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.4
[pip3] pytorch-fid==0.3.0
[pip3] pytorch-lightning==1.4.2
[pip3] torch==2.0.1
[pip3] torch-ema==0.3
[pip3] torch-fidelity==0.3.0
[pip3] torchaudio==2.0.2
[pip3] torcheval==0.0.7
[pip3] torchmetrics==0.5.0
[pip3] torchvision==0.15.2
[pip3] triton==2.0.0
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h2bc3f7f_2
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-service 2.4.0 py38h7f8727e_0
[conda] mkl_fft 1.3.1 py38hd3c417c_0
[conda] mkl_random 1.2.2 py38h51133e4_0
[conda] numpy 1.24.4 pypi_0 pypi
[conda] pytorch-fid 0.3.0 pypi_0 pypi
[conda] pytorch-lightning 1.4.2 pypi_0 pypi
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.0.1 pypi_0 pypi
[conda] torch-ema 0.3 pypi_0 pypi
[conda] torch-fidelity 0.3.0 pypi_0 pypi
[conda] torchaudio 2.0.2 pypi_0 pypi
[conda] torcheval 0.0.7 pypi_0 pypi
[conda] torchmetrics 0.5.0 pypi_0 pypi
[conda] torchvision 0.15.2 pypi_0 pypi
[conda] triton 2.0.0 pypi_0 pypi

[AwalkZY]

Same here. Strongly suggest maintainers aligning the calculation with pytorch-fid or torch-fidelity.

[Tiphaine-Lela]

I got the same problem. Increasing the precision to float64 is needed.