Make manual optimization mandatory for multiple optimizers

[awaelchli]

What does this PR do?

WARNING Estimated time to review: 2 hours + 🤣.

This PR reworks how optimization with multiple optimizers works.

Closes #12169

Before

User returns multiple optimizers from the configure_optimizers hook
The training step gets a optimizer_idx argument and gets called as many times with the same batch as there are optimizers.

Pro:

• User doesn't have to write the zero_grad and step calls
• Toggling of optimizers handled automatically

Cons:

• Users struggle to understand how it works under the hood
• Stepping optimizers at different intervals is possible, but cumbersome to configure
• Loop implementation is complex, hard to follow and debug for contributors
• Non-standard optimization requires the user to override several hooks (backward, optimizer_step) etc. which increases the surface for bugs (simplest example is GAN).
• Complex output post-processing for the batch-end and epoch-end hooks required (lists of lists of lists)

Now

Returning multiple optimizers from configure_optimizers hook will raise an error. You need to set self.automatic_optimization=False and implement manual optimization in your training step.

Pro:

• Non-standard optimization is more intuitive to implement (simplest example is GAN)
• Stepping optimizers at custom intervals is easy to understand and debug because logic is inlined in training step by user
• Loops become simpler
• No special outputs processing needed for batch-end and epoch-end hooks

Cons:

• User has to call the zero_grad and step and manual_backward manually.
• Gradient accumulation is not supported out of the box for manual optimization. You have to accumulate yourself. For the highest flexibility, Fabric should be considered.

Does your PR introduce any breaking changes? If yes, please list them.

• Removed opt_idx argument from BaseFinetuning.finetune_function callback method
• Removed opt_idx argument from Callback.on_before_optimizer_step callback method
• Removed optimizer_idx as an optional argument in LightningModule.training_step
• Removed optimizer_idx argument from LightningModule.on_before_optimizer_step
• Removed optimizer_idx argument from LightningModule.configure_gradient_clipping
• Removed optimizer_idx argument from LightningModule.optimizer_step
• Removed optimizer_idx argument from LightningModule.optimizer_zero_grad
• Removed optimizer_idx argument from LightningModule.lr_scheduler_step
• Removed support for declaring optimizer frequencies in the dictionary returned from LightningModule.configure_optimizers
• Removed arguments optimizer and optimizer_idx from LightningModule.backward
• Removed optimizer_idx argument from PrecisionPlugin.optimizer_step and all of its overrides in subclasses
• Removed optimizer_idx argument from PrecisionPlugin.{optimizer_step,backward} and all of its overrides in subclasses
• Removed optimizer_idx argument from Strategy.{optimizer_step,backward} and all of its overrides in subclasses
• Removed Trainer.optimizer_frequencies attribute

Follow ups to this PR:

• Documentation updates
• Demote optimizer loop class from being a loop

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @justusschock @awaelchli @Borda @carmocca

[carmocca]

LGTM. Huge effort!

[Borda]

quite long but over all looks good 👍