Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bitsandbytes layer conversion under init_module context manager #18914

Merged
merged 6 commits into from
Nov 1, 2023

Conversation

awaelchli
Copy link
Contributor

@awaelchli awaelchli commented Nov 1, 2023

What does this PR do?

Fixes #18913

This test fails on master, but didn't run in our CI because GPU CI does not install bitsandbytes:
https://github.com/Lightning-AI/lightning/blob/7a5b7f5561b34f13da1e0aa26af18b94b62494cc/tests/tests_fabric/plugins/precision/test_bitsandbytes.py#L102

This PR adds the missing installation. Now the test passes:

https://dev.azure.com/Lightning-AI/72ab7ed8-b00f-4b6e-b131-3388f7ffafa7/_build/results?buildId=180654&view=logs&j=3f274fac-2e11-54ca-487e-194c91f3ae9f

plugins/precision/test_bitsandbytes.py::test_bitsandbytes_layers[args0-expected0] PASSED [ 37%]
plugins/precision/test_bitsandbytes.py::test_bitsandbytes_layers[args1-expected1] PASSED [ 38%]

📚 Documentation preview 📚: https://pytorch-lightning--18914.org.readthedocs.build/en/18914/

cc @carmocca @Borda @justusschock @awaelchli

@github-actions github-actions bot added the fabric lightning.fabric.Fabric label Nov 1, 2023
@carmocca carmocca added this to the 2.1.x milestone Nov 1, 2023
@awaelchli
Copy link
Contributor Author

What would be the best way to install bitsandbytes in the GPU CI @carmocca @Borda ? Should I put it into the workflow or have it in the docker image?

@carmocca
Copy link
Contributor

carmocca commented Nov 1, 2023

I think the docker image. We could put it in extra.txt but then users might have issues if it fails to install in certain platform (windows? macos?)

@github-actions github-actions bot added ci Continuous Integration dependencies Pull requests that update a dependency file dockers labels Nov 1, 2023
@awaelchli awaelchli changed the title Fix bitsandbytes layer conversion under init_module context manager [WIP] Fix bitsandbytes layer conversion under init_module context manager Nov 1, 2023
@awaelchli awaelchli marked this pull request as ready for review November 1, 2023 14:56
Copy link
Contributor

github-actions bot commented Nov 1, 2023

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success
pl-cpu (macOS-11, lightning, 3.9, 1.12) success
pl-cpu (macOS-11, lightning, 3.10, 1.13) success
pl-cpu (macOS-11, lightning, 3.10, 2.0) success
pl-cpu (macOS-11, lightning, 3.10, 2.1) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success
pl-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success
pl-cpu (windows-2022, lightning, 3.9, 1.12) success
pl-cpu (windows-2022, lightning, 3.10, 1.13) success
pl-cpu (windows-2022, lightning, 3.10, 2.0) success
pl-cpu (windows-2022, lightning, 3.10, 2.1) success
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success
pl-cpu (macOS-12, pytorch, 3.11, 2.0) success
pl-cpu (macOS-12, pytorch, 3.11, 2.1) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1) success
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success
pl-cpu (windows-2022, pytorch, 3.11, 2.1) success

These checks are required after the changes to src/lightning/fabric/strategies/strategy.py, requirements/pytorch/extra.txt.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) (testing Lightning | latest) success
pytorch-lightning (GPUs) (testing PyTorch | latest) success

These checks are required after the changes to requirements/pytorch/extra.txt, src/lightning/fabric/strategies/strategy.py.

🟢 pytorch_lightning: Benchmarks
Check ID Status
lightning.Benchmarks success

These checks are required after the changes to requirements/pytorch/extra.txt, src/lightning/fabric/strategies/strategy.py.

🟢 fabric: Docs
Check ID Status
docs-make (fabric, doctest) success
docs-make (fabric, html) success

These checks are required after the changes to src/lightning/fabric/strategies/strategy.py.

🟢 pytorch_lightning: Docs
Check ID Status
docs-make (pytorch, doctest) success
docs-make (pytorch, html) success

These checks are required after the changes to requirements/pytorch/extra.txt.

🟢 pytorch_lightning: Docker
Check ID Status
build-cuda (3.9, 1.12, 11.7.1) success
build-cuda (3.9, 1.13, 11.8.0) success
build-cuda (3.9, 1.13, 12.0.1) success
build-cuda (3.10, 2.0, 11.8.0) success
build-cuda (3.10, 2.1, 12.1.0) success
build-pl (3.9, 1.12, 11.7.1) success
build-pl (3.9, 1.13, 11.8.0) success
build-pl (3.9, 1.13, 12.0.1) success
build-pl (3.10, 2.0, 11.8.0) success
build-pl (3.10, 2.1, 12.1.0) success

These checks are required after the changes to requirements/pytorch/extra.txt.

🟢 lightning_fabric: CPU workflow
Check ID Status
fabric-cpu (macOS-11, lightning, 3.8, 1.12, oldest) success
fabric-cpu (macOS-11, lightning, 3.9, 1.12) success
fabric-cpu (macOS-11, lightning, 3.10, 1.13) success
fabric-cpu (macOS-11, lightning, 3.10, 2.0) success
fabric-cpu (macOS-11, lightning, 3.11, 2.1) success
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.12, oldest) success
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12) success
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0) success
fabric-cpu (ubuntu-20.04, lightning, 3.11, 2.1) success
fabric-cpu (windows-2022, lightning, 3.8, 1.12, oldest) success
fabric-cpu (windows-2022, lightning, 3.9, 1.12) success
fabric-cpu (windows-2022, lightning, 3.10, 1.13) success
fabric-cpu (windows-2022, lightning, 3.10, 2.0) success
fabric-cpu (windows-2022, lightning, 3.11, 2.1) success
fabric-cpu (macOS-11, fabric, 3.8, 1.13) success
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13) success
fabric-cpu (windows-2022, fabric, 3.8, 1.13) success
fabric-cpu (macOS-12, fabric, 3.11, 2.0) success
fabric-cpu (macOS-12, fabric, 3.11, 2.1) success
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.0) success
fabric-cpu (ubuntu-22.04, fabric, 3.11, 2.1) success
fabric-cpu (windows-2022, fabric, 3.11, 2.0) success
fabric-cpu (windows-2022, fabric, 3.11, 2.1) success

These checks are required after the changes to src/lightning/fabric/strategies/strategy.py.

🟢 lightning_fabric: Azure GPU
Check ID Status
lightning-fabric (GPUs) (testing Fabric | latest) success
lightning-fabric (GPUs) (testing Lightning | latest) success

These checks are required after the changes to src/lightning/fabric/strategies/strategy.py.

🟢 mypy
Check ID Status
mypy success

These checks are required after the changes to requirements/pytorch/extra.txt, src/lightning/fabric/strategies/strategy.py.

🟢 install
Check ID Status
install-pkg (ubuntu-22.04, app, 3.8) success
install-pkg (ubuntu-22.04, app, 3.11) success
install-pkg (ubuntu-22.04, fabric, 3.8) success
install-pkg (ubuntu-22.04, fabric, 3.11) success
install-pkg (ubuntu-22.04, pytorch, 3.8) success
install-pkg (ubuntu-22.04, pytorch, 3.11) success
install-pkg (ubuntu-22.04, lightning, 3.8) success
install-pkg (ubuntu-22.04, lightning, 3.11) success
install-pkg (ubuntu-22.04, notset, 3.8) success
install-pkg (ubuntu-22.04, notset, 3.11) success
install-pkg (macOS-12, app, 3.8) success
install-pkg (macOS-12, app, 3.11) success
install-pkg (macOS-12, fabric, 3.8) success
install-pkg (macOS-12, fabric, 3.11) success
install-pkg (macOS-12, pytorch, 3.8) success
install-pkg (macOS-12, pytorch, 3.11) success
install-pkg (macOS-12, lightning, 3.8) success
install-pkg (macOS-12, lightning, 3.11) success
install-pkg (macOS-12, notset, 3.8) success
install-pkg (macOS-12, notset, 3.11) success
install-pkg (windows-2022, app, 3.8) success
install-pkg (windows-2022, app, 3.11) success
install-pkg (windows-2022, fabric, 3.8) success
install-pkg (windows-2022, fabric, 3.11) success
install-pkg (windows-2022, pytorch, 3.8) success
install-pkg (windows-2022, pytorch, 3.11) success
install-pkg (windows-2022, lightning, 3.8) success
install-pkg (windows-2022, lightning, 3.11) success
install-pkg (windows-2022, notset, 3.8) success
install-pkg (windows-2022, notset, 3.11) success

These checks are required after the changes to src/lightning/fabric/strategies/strategy.py, requirements/pytorch/extra.txt.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

@github-actions github-actions bot added pl Generic label for PyTorch Lightning package and removed ci Continuous Integration dockers labels Nov 1, 2023
Copy link

codecov bot commented Nov 1, 2023

Codecov Report

Merging #18914 (524226b) into master (7a5b7f5) will decrease coverage by 49%.
The diff coverage is 100%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #18914      +/-   ##
==========================================
- Coverage      75%      27%     -49%     
==========================================
  Files         448      440       -8     
  Lines       35910    35758     -152     
==========================================
- Hits        27066     9569   -17497     
- Misses       8844    26189   +17345     

@awaelchli awaelchli added the bug Something isn't working label Nov 1, 2023
@awaelchli awaelchli changed the title [WIP] Fix bitsandbytes layer conversion under init_module context manager Fix bitsandbytes layer conversion under init_module context manager Nov 1, 2023
@awaelchli
Copy link
Contributor Author

awaelchli commented Nov 1, 2023

The docker build passed, I pushed the image to the hub, re-triggered the GPU CI and the test passes now!

requirements/pytorch/extra.txt Show resolved Hide resolved
@mergify mergify bot added the ready PRs ready to be merged label Nov 1, 2023
@awaelchli awaelchli merged commit 13e1926 into master Nov 1, 2023
127 of 129 checks passed
@awaelchli awaelchli deleted the bugfix/bnb-init branch November 1, 2023 18:10
Borda pushed a commit that referenced this pull request Nov 2, 2023
Borda pushed a commit that referenced this pull request Nov 2, 2023
Borda pushed a commit that referenced this pull request Nov 2, 2023
lantiga pushed a commit that referenced this pull request Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dependencies Pull requests that update a dependency file fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bitsandbytes doesn't convert layers under init_module context
3 participants