Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rank_zero_only decorator for LSF environments #12587

Merged
merged 7 commits into from
Apr 7, 2022

Conversation

awaelchli
Copy link
Contributor

@awaelchli awaelchli commented Apr 3, 2022

What does this PR do?

Fixes #12583

The rank_zero_only decorator accesses a global variable to determine the rank. This is so because the decorator has no access to the trainer. When a logger makes a call to self.experiment before the trainer is instantiated (and the cluster environment with global_rank etc. is available), the rank must be determined from the environment variables directly. So far, only the canonical RANK and slurm variables were checked. Since LSF is supported directly by Lightning, we should include the environment variable in the global rank utility.

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @awaelchli

@awaelchli awaelchli added bug Something isn't working environment: lsf labels Apr 3, 2022
@awaelchli awaelchli changed the base branch from master to tests/enable-rank-zero April 3, 2022 04:04
@awaelchli awaelchli added this to the 1.6.x milestone Apr 3, 2022
@awaelchli awaelchli marked this pull request as ready for review April 3, 2022 23:07
@awaelchli awaelchli marked this pull request as draft April 3, 2022 23:11
Base automatically changed from tests/enable-rank-zero to master April 4, 2022 14:24
@awaelchli awaelchli marked this pull request as ready for review April 4, 2022 14:34
@awaelchli awaelchli changed the title Fix rank_zero_only decorator for LSF environments Update rank_zero_only decorator for LSF environments Apr 4, 2022
@mergify mergify bot added the ready PRs ready to be merged label Apr 4, 2022
@awaelchli awaelchli enabled auto-merge (squash) April 4, 2022 19:58
@mergify mergify bot added has conflicts and removed ready PRs ready to be merged labels Apr 6, 2022
@mergify mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Apr 7, 2022
@awaelchli awaelchli merged commit 6aa8e26 into master Apr 7, 2022
@awaelchli awaelchli deleted the bugfix/rank-zero-lsf branch April 7, 2022 10:46
@rohitgr7 rohitgr7 mentioned this pull request Jul 1, 2022
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working environment: lsf ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrading from 1.5.10 to 1.6.0 causes the PyTorch WandB logger to produce multiple experiments
6 participants