Manual vs Automatic Optimization Discrepancy in VAE Models

[TTK95]

I've encountered an intriguing phenomenon while working with Variational Autoencoder (VAE) models trained using Lightning on PyTorch. Specifically, I've noticed significant differences in the loss curves when using manual optimization compared to automatic optimization.

To provide some context, I have two VAE models and I've attached a screenshot showcasing the distinct loss curves for both cases. This has piqued my curiosity and I'm eager to dive deeper into understanding the underlying mechanics driving this difference.

Here's what I'd like to discuss:

• Has anyone else experienced a similar phenomenon in their VAE models or other types of generative models or in general?
• What could be the potential factors contributing to this discrepancy between manual and automatic optimization?
• Are there any recommended practices or insights for handling manual optimization in VAE models to achieve more consistent results?

My opt_step code snippet in the training_step func:

vae_opt.zero_grad()
self.manual_backward(loss)
vae_opt.step()

loss is calculated in both models the same way!

Thank you in advance for your time and assistance.

[mustafa-tavus]

hi, I'm wondering if you were able to solve this issue?