Batch size dependent FID

[nicolas-dufour]

🐛 Bug

As pointed out by @kimihailv in #1198, FID seem to be batch-size dependent.
After some experimentation, the dependency seems to be linked to the inception network.

To Reproduce

If one runs:

inception = NoTrainInceptionV3(name="inception-v3-compat", features_list=["2048"])
inception.to(device)
imgs = torch.randn(100, 3, 256, 256).to(device)
imgs = ((imgs.clamp(-1, 1) / 2 + 0.5) * 255).to(torch.uint8)
features = []
for img in imgs:
    feature = inception(img.unsqueeze(0))
    features.append(feature)
features_b_1 = torch.cat(features, dim=0)
features = inception(imgs)

We then have:

torch.allclose(features, features_b_1)
>> False
torch.norm(features - features_b_1, p=2)
>> tensor(0.2950, device='cuda:0')

To be noted that when replacing

features = inception(imgs)

by

features = []
for img in imgs:
    feature = inception(img.unsqueeze(0))
    features.append(feature)
features= torch.cat(features, dim=0)

the FID is again batch independent. However, this is not a possible fix. Indeed, first this is very ineficient. Also, from experimentation, the batch bias in FID seems to be higher from small batch-sizes. If we compute FID between 2 uniformly sampled distributions with a 1000 points each, if we compute it with a batch size of 1000, we get FID 1.9 but if we compute it with batch-size=2, then the FID is 10. Since we sample from the same distribution, FID should be as close too zero as possible.

Expected behavior

FID computation should be batch_size independent

Environment

• TorchMetrics version (and how you installed TM, e.g. conda, pip, build from source): 0.10.3
• Python & PyTorch Version (e.g., 1.0): 3.10 and 1.13
• Any other relevant information such as OS (e.g., Linux): Linux

[SkafteNicki]

Hi @nicolas-dufour, thanks for reporting this issue.
I tried reproducing your results, but were unable to do that, please see this notebook:
https://colab.research.google.com/drive/16aVuIJQj7TUmhiy6TGL2QcyQOTb3ibJr?usp=sharing
On CPU i get a norm difference of tensor(0.0002) and on CUDA i get tensor(0.0172) between the two approaches, which seems reasonable to still calculate the metric correctly.

[nicolas-dufour]

Hi @SkafteNicki, thanks for checking this out.

Hum that is strange! From further experiments, i found that the discrepancy disappeared when using float64. Maybe the problem is accelerator dependent? The previous experiment was done on a RTX 3090.

Also, I've observed that the impact on FID was minimal for dataset size > 100. However, it's still weird that the metric changes with respect to the batch size. One solution would be to offer the option to run the embedding network at float64 precision.

[SkafteNicki]

Hi again @nicolas-dufour,
So I created PR #1628 that will allow the user to run the embedding network with float64 by simply calling the .set_dtype method of the metric.

from torchmetrics.image.fid import NoTrainInceptionV3, FrechetInceptionDistance
import torch

metric = FrechetInceptionDistance()
metric.set_dtype(torch.float64)

imgs = torch.randn(1, 3, 256, 256)
imgs = ((imgs.clamp(-1, 1) / 2 + 0.5) * 255).to(torch.uint8)

metric.inception(imgs)

still need a bit of testing a documentation.