GAN tasks in Flash

[aniketmaurya]

🚀 Feature

Support for GAN models - image2image translation, image generation.

Motivation

Flash already has text, vision, video, and tabular. It will be awesome to have a module for GANs too! We can start with models already present in pl_bolts and another repository like lucidrains

PS: I can work on implementing this feature in Flash!

[ethanwharris]

Yeah, this would be really cool, can maybe share some components with the pystiche integration being added in #262 😃

[edgarriba]

Should we group all those those tasks under image_generation ? basically to cover all those that inputs/outputs images /cc @pmeier

[pmeier]

Should we group all those those tasks under image_generation ?

IMO that is a good idea, but image_generation is not general enough: for example style transfer or cycle GANs do not generate image but perform an image translation.

[aniketmaurya]

Agree @pmeier . I would suggest to have a Task for GAN and subtasks can be image2image translation, image Generation, image to text Generation, etc.

Inside image translation also we have multiple category - paired and unpaired image2image translation

[edgarriba]

@aniketmaurya My point was more that to me, a GAN is a technique to solve a Task but not a task itself.

[aniketmaurya]

I created a quick and dirty poc for gan module in Flash. Pls have a look and let me know if I am proceeding right or not

https://github.com/aniketmaurya/lightning-flash/blob/gan_poc/flash/gan/image_translation/models/paired_translation.py

[edgarriba]

@aniketmaurya I suggest to go for one task at a time and create just flash.vision.image_translation - the task should be just ImageTranslation.

You can check on other tasks like segmentation and try to follow the same structure of files (think as each task a mini-project inside flash):

• backbones.py
• data.py
• model.py
• serialisation.py
• transforms.py

In this particular case, I would do something like backbone = pix2pix (or similar known method) and then make somehow a PairedImageTranslationBackbone class that handles the generator and discriminator during the forward pass.

Regarding, the training - do you have any implementation in mind yet ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.