Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for torch.float weighted networks for FID and KID calculations. #2483

Merged
merged 28 commits into from
Apr 16, 2024

Conversation

furkan-celik
Copy link
Contributor

@furkan-celik furkan-celik commented Mar 31, 2024

What does this PR do?

For evaluation of generated images in terms of how similar looking they are to the real images metrics FID is heavily used. However as pointed out in several papers, FID has some inherent flaws one of which is ImageNet InceptionV3 model is not suitable for all domains. Domains like medical MRI might be different enough to ImageNet that activation functions are not relevant to make FID comparison. For cases like this different domains use different variants of InceptionV3 model as feature extractor.

However, current implementation of FID and KID in torchmetrics only supports models with byteTensor weights as it is optimized for the tochmetrics' InceptionV3 model. In this PR I have given a bit more control to the developer so that it is possible to have better support for custom feature extractors with different dtypes and input sizes.

Details of this issue can be found in the following paper:
Liu, Shaohui, et al. "An improved evaluation framework for generative adversarial networks." arXiv preprint arXiv:1803.07474 (2018).

Fixes #<issue_number>

Before submitting
  • [X I had issues with my implementations and didn't found a solution] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • [+] Did you read the contributor guideline, Pull Request section?
  • [+] Did you make sure to update the docs?
  • [+] Did you write any new necessary tests?
PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃


📚 Documentation preview 📚: https://torchmetrics--2483.org.readthedocs.build/en/2483/

Copy link

codecov bot commented Mar 31, 2024

Codecov Report

Merging #2483 (51dcc91) into master (5259c22) will decrease coverage by 31%.
The diff coverage is 0%.

Additional details and impacted files
@@           Coverage Diff            @@
##           master   #2483     +/-   ##
========================================
- Coverage      69%     38%    -31%     
========================================
  Files         307     307             
  Lines       17404   17410      +6     
========================================
- Hits        11989    6662   -5327     
- Misses       5415   10748   +5333     

@mergify mergify bot added the ready label Apr 10, 2024
@Borda Borda changed the title Support for torch.float weighted networks for FID and KID calculations. Support for torch.float weighted networks for FID and KID calculations. Apr 12, 2024
@mergify mergify bot removed the ready label Apr 12, 2024
Copy link
Member

@SkafteNicki SkafteNicki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good :)

src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
src/torchmetrics/image/fid.py Outdated Show resolved Hide resolved
furkan-celik and others added 2 commits April 12, 2024 17:41
Co-authored-by: Nicki Skafte Detlefsen <[email protected]>
Co-authored-by: Nicki Skafte Detlefsen <[email protected]>
@mergify mergify bot added the ready label Apr 13, 2024
@SkafteNicki SkafteNicki enabled auto-merge (squash) April 15, 2024 10:59
@SkafteNicki SkafteNicki merged commit 822dba2 into Lightning-AI:master Apr 16, 2024
61 of 62 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants