Remove is_global_zero check in training_epoch_loop

[daniellepintz]

What does this PR do?

This check in training_epoch_loop is unnecessary so we can remove it.

This is because save on the LightningLoggerBase API is not implemented:
https://github.com/PyTorchLightning/pytorch-lightning/blob/b29b07e9788311326bca4779d70e89eb36bfc57f/pytorch_lightning/loggers/base.py#L173-L174

And save is only overridden for CSV and TB loggers, and both implementations are decorated with @rank_zero_only:
https://github.com/PyTorchLightning/pytorch-lightning/blob/b29b07e9788311326bca4779d70e89eb36bfc57f/pytorch_lightning/loggers/csv_logs.py#L204-L207
https://github.com/PyTorchLightning/pytorch-lightning/blob/b29b07e9788311326bca4779d70e89eb36bfc57f/pytorch_lightning/loggers/tensorboard.py#L254-L264

Fixes #12127

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[justusschock]

This might break third-party loggers whose integration is not part of the PL codebase though and who didn't decorate the save function but still implement it.
Personally, I think it makes more sense to have an explicit check in the loop, than to have every logger implementation taking care of that.

Also: is it documented somewhere, that this is required?

[tchaton]

LGTM ! But I still see value in @justusschock comment. This would have to be explicit in the documentation.

[awaelchli]

ok with me as an intermediate step for #11676 (would address the comment of @justusschock and shift the responsibility to the log method directly)

[ananthsub]

@justusschock I agree, this is a breaking change. @daniellepintz could you update the changelog to document the behavior change here?

The motivation is the same as #8589 . In this case, users may want to have a logger per process to publish metrics. This can be very useful in distributed training for detecting outliers.

Currently, the loop is hardcoded to only flush the metrics on rank 0. In v1.7 this behavior will change given flush_logs_every_n_steps will be removed from the Trainer, since loggers will take over more the responsibility (on what ranks to log + how frequently to flush). Between v1.6 and v1.7 though we can still offer more flexibility through this PR.

[daniellepintz]

I updated the changelog!