How to train GAN with DDP without setting "find_unused_parameters=True"

[function2-llx]

Hello everyone,

I am developing a GAN using Lightning and Distributed Data Parallel (DDP) for training. Due to the alternating training of the generator and discriminator in GANs, fulfilling DDP's requirement of all model parameters contributing to the loss calculation every forward pass presents a challenge.

While setting find_unused_parameters=True in the DDP configuration ensures successful execution, it compromises performance. Consequently, I want to know if it's possible to train GANs with DDP without setting find_unused_parameters=True.

Any insights or recent developments related to this issue would be much appreciated. Thank you in advance!

[function2-llx]

Well, I find that using fabric might be a good idea, since you can wrap generator and discriminator in DDP respectively.

[Feynman1999]

have you solved it?

[function2-llx]

Sadly, no. Besides this one, there are just tremendous amount of issues using manual optimization with PyTorch Lightning. My recommendation is that don't ever try it, not to mention that currently the development of this library is significantly slowed down and I don't think these issues will be properly addressed. Use fabric or other libraries.

[mustafa-tavus]

hey I'm also facing issues with manual optimization regarding the loss curves and validations results compared to automatic optimization. Have you encountered similar issues?