Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Patch for device placement (Reduce host and device syncs) #17334

Merged
merged 21 commits into from
Apr 25, 2023
Merged

Patch for device placement (Reduce host and device syncs) #17334

merged 21 commits into from
Apr 25, 2023

Conversation

shanmugamr1992
Copy link
Contributor

What does this PR do?

Fixes #<issue_number>

Before submitting
  • Was this discussed/agreed via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist
  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Apr 11, 2023
@shanmugamr1992 shanmugamr1992 changed the title Patch for device placement Patch for device placement (Reduce host and device syncs) Apr 14, 2023
@shanmugamr1992
Copy link
Contributor Author

The main idea is to reduce host and device sync.
We also want to use CUDA graphs with DDP. So adding necessary changes.

@carmocca carmocca added logging Related to the `LoggerConnector` and `log()` performance community This PR is from the community labels Apr 15, 2023
@carmocca carmocca added this to the 2.1 milestone Apr 15, 2023
@carmocca carmocca self-assigned this Apr 15, 2023
@carmocca carmocca added the breaking change Includes a breaking change label Apr 15, 2023
@Borda Borda requested a review from carmocca April 18, 2023 10:56
@github-actions github-actions bot added the fabric lightning.fabric.Fabric label Apr 24, 2023
@mergify mergify bot added the ready PRs ready to be merged label Apr 24, 2023
@carmocca carmocca enabled auto-merge (squash) April 25, 2023 00:47
@carmocca carmocca merged commit 524608c into Lightning-AI:master Apr 25, 2023
awaelchli
awaelchli reviewed Apr 27, 2023
View reviewed changes
src/lightning/pytorch/strategies/ddp.py
Comment on lines +186 to +187
# https://pytorch.org/docs/stable/notes/cuda.html#id5
ctx = torch.cuda.stream(torch.cuda.Stream()) if torch.cuda.is_available() else nullcontext()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we avoid using the stream if not running on CUDA? Even if there are no known side effects, for sanity I would prefer that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, yes. We can check the root device type instead

@awaelchli awaelchli mentioned this pull request Apr 27, 2023
Merged
@awaelchli awaelchli mentioned this pull request Nov 27, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@carmocca carmocca carmocca approved these changes

@awaelchli awaelchli awaelchli left review comments

@Borda Borda Borda approved these changes

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@tchaton tchaton Awaiting requested review from tchaton tchaton is a code owner

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

Assignees

@carmocca carmocca

Labels
breaking change Includes a breaking change community This PR is from the community fabric lightning.fabric.Fabric logging Related to the `LoggerConnector` and `log()` performance pl Generic label for PyTorch Lightning package ready PRs ready to be merged
Projects
None yet
Milestone
2.1
Development

Successfully merging this pull request may close these issues.
4 participants
@shanmugamr1992 @awaelchli @Borda @carmocca