Split train- and val progress into separate bars #16695

awaelchli · 2023-02-08T23:05:16Z

What does this PR do?

A long pending request from the community is an improved progress bar. So far, the training and validation counts were always added together in one progress bar. This has the drawback that more often than not, the time estimate for completion is wrong, because the validation loop typically runs much faster than the regular training loop. Furthermore, users were often confused about the total iteration count, because they didn't realize that the total number was train + val batches. See #6735, #1121, #6406

This PR splits the bars into two separate ones. The validation bar appears and disappears as needed as before. The PR introduces the change for both TQDM and Rich.

Video recording with 10 train batches 5 val batches, validation every 2 train batches:

TQDM:

Screen.Recording.2023-02-09.at.00.11.26.mov

Rich:

Screen.Recording.2023-02-09.at.00.12.49.mov

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

cc @Borda @justusschock @awaelchli

for more information, see https://pre-commit.ci

… feature/split-progress-bar

github-actions · 2023-02-08T23:18:44Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, tests/tests_pytorch/callbacks/progress/test_base_progress.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, tests/tests_pytorch/callbacks/progress/test_base_progress.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, tests/tests_pytorch/callbacks/progress/test_base_progress.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, tests/tests_pytorch/callbacks/progress/test_base_progress.py, tests/tests_pytorch/callbacks/progress/test_rich_progress_bar.py, tests/tests_pytorch/callbacks/progress/test_tqdm_progress_bar.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py, docs/source-pytorch/common/progress_bar.rst.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/pytorch/callbacks/progress/base.py, src/lightning/pytorch/callbacks/progress/rich_progress.py, src/lightning/pytorch/callbacks/progress/tqdm_progress.py.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

src/lightning/pytorch/callbacks/progress/rich_progress.py

src/lightning/pytorch/callbacks/progress/tqdm_progress.py

docs/source-pytorch/common/progress_bar.rst

src/lightning/pytorch/CHANGELOG.md

src/lightning/pytorch/callbacks/progress/tqdm_progress.py

Borda

Just curious do we have also a bar for testing and predictions?

awaelchli · 2023-02-09T18:07:03Z

@Borda These are already there and remain unchanged. They were never coupled to the "main" bar, so no changes there.

awaelchli added 2 commits February 8, 2023 23:47

split tqdm bar

f784f96

update rich bar

bd07a8b

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 8, 2023

awaelchli added this to the 2.0 milestone Feb 8, 2023

awaelchli self-assigned this Feb 8, 2023

[pre-commit.ci] auto fixes from pre-commit.com hooks

212b494

for more information, see https://pre-commit.ci

github-actions bot added the pl Generic label for PyTorch Lightning package label Feb 8, 2023

awaelchli added 3 commits February 9, 2023 00:15

update test

45164c0

Merge remote-tracking branch 'origin/feature/split-progress-bar' into…

cfa7ea4

… feature/split-progress-bar

changelog

fe411ce

awaelchli marked this pull request as ready for review February 8, 2023 23:18

awaelchli requested review from carmocca, justusschock and williamFalcon as code owners February 8, 2023 23:18

remove redundant test

21f6f19

carmocca reviewed Feb 8, 2023

View reviewed changes

src/lightning/pytorch/callbacks/progress/rich_progress.py Outdated Show resolved Hide resolved

src/lightning/pytorch/callbacks/progress/rich_progress.py Show resolved Hide resolved

src/lightning/pytorch/callbacks/progress/tqdm_progress.py Outdated Show resolved Hide resolved

awaelchli added 2 commits February 9, 2023 00:54

update docs

1781ba6

update docs

cf75940

awaelchli requested review from edenlightning, lantiga and Borda as code owners February 9, 2023 00:01

main -> train

0c06145

carmocca approved these changes Feb 9, 2023

View reviewed changes

docs/source-pytorch/common/progress_bar.rst Outdated Show resolved Hide resolved

src/lightning/pytorch/CHANGELOG.md Show resolved Hide resolved

src/lightning/pytorch/callbacks/progress/tqdm_progress.py Outdated Show resolved Hide resolved

awaelchli added 3 commits February 9, 2023 18:47

Merge branch 'master' into feature/split-progress-bar

c19f925

left-over renames

5f193fc

changelog

bdccf1a

justusschock approved these changes Feb 9, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Feb 9, 2023

Borda approved these changes Feb 9, 2023

View reviewed changes

awaelchli enabled auto-merge (squash) February 9, 2023 18:19

awaelchli merged commit 18106a8 into master Feb 9, 2023

awaelchli deleted the feature/split-progress-bar branch February 9, 2023 19:43

awaelchli mentioned this pull request Mar 15, 2023

Apply ProgressBar Refresh among all epoch and not across train and validation. #11342

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split train- and val progress into separate bars #16695

Split train- and val progress into separate bars #16695

awaelchli commented Feb 8, 2023 •

edited

Loading

github-actions bot commented Feb 8, 2023 •

edited

Loading

Borda left a comment

awaelchli commented Feb 9, 2023

Split train- and val progress into separate bars #16695

Split train- and val progress into separate bars #16695

Conversation

awaelchli commented Feb 8, 2023 • edited Loading

What does this PR do?

Before submitting

PR review

Did you have fun?

github-actions bot commented Feb 8, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

Borda left a comment

Choose a reason for hiding this comment

awaelchli commented Feb 9, 2023

awaelchli commented Feb 8, 2023 •

edited

Loading

github-actions bot commented Feb 8, 2023 •

edited

Loading