-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3/n inter batch parallelism #9052
Conversation
for more information, see https://pre-commit.ci
…rchLightning/pytorch-lightning into 1/n_inter_batch_parallelism
…lightning into 0/n_inter_batch_parallelism
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…rchLightning/pytorch-lightning into 3/n_inter_batch_parallelism
…rchLightning/pytorch-lightning into 3/n_inter_batch_parallelism
for more information, see https://pre-commit.ci
def training_step(self, dataloader_iter: Iterator, *args) -> STEP_OUTPUT: | ||
output = super().training_step(dataloader_iter) | ||
if self.trigger_stop_iteration and args[0] == EXPECT_NUM_BATCHES_PROCESSED: | ||
raise StopIteration |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tchaton @awaelchli Is using a StopIteration
here international or was it just meant as a DummyException
?
This has the problem that it's the exception checked by the loops to stop, so after it is raised, the loops will try to run on_run_end
but any further state done at the end of advance
or on_advance_end
will not have happened so things can be out of sync.
Blocking #9349
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is intentional, but it should work as expected.
What does this PR do?
Blocked by #9047
This PR adds remove entirely InterBatchProcessor and rely on DataFetcher.
fixes #8867
fixes #8316
Does your PR introduce any breaking changes? If yes, please list them.
Before submitting
PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:
Did you have fun?
Make sure you had fun coding 🙃