[App] Refactor cloud dispatch and update to new API #16456

ethanwharris · 2023-01-20T20:56:36Z

What does this PR do?

Updates cloud dispatch to the new API
breaks down the large method into several smaller ones and calls them in a consistent order (e.g. so we don't raise errors after things have started to happen)
adds the option to export LIGHTNING_CLOUD_PRINT_SPECS=gallery or LIGHTNING_CLOUD_PRINT_SPECS=web to generate app specs for use in the gallery or web dispatch

Note: Tests left deliberately as unchanged as possible to ensure that the core logic hasn't changed.

Fixes #15012

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda

github-actions · 2023-01-20T20:57:06Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, app, 3.8, latest)	success	✅
app-pytest (macOS-11, app, 3.8, oldest)	success	✅
app-pytest (macOS-11, lightning, 3.9, latest)	success	✅
app-pytest (ubuntu-20.04, app, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, app, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅
app-pytest (windows-2022, app, 3.8, oldest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅

These checks are required after the changes to src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py, tests/tests_app/cli/test_cloud_cli.py, tests/tests_app/runners/test_cloud.py, tests/tests_app/utilities/test_app_helpers.py, requirements/app/base.txt.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (macOS-11, app, 3.9, oldest)	success	✅
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅
app-examples (windows-2022, app, 3.9, oldest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅

These checks are required after the changes to src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py, tests/integrations_app/public/test_commands_and_api.py, requirements/app/base.txt.

🟢 lightning_app: Azure

Check ID	Status
App.cloud-e2e	success	✅

These checks are required after the changes to src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py, requirements/app/base.txt, tests/integrations_app/public/test_commands_and_api.py.

🟢 lightning_app: Docs

Check ID	Status
make-doctest (app)	success	✅
make-html (app)	success	✅

These checks are required after the changes to src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py, requirements/app/base.txt.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/app/base.txt, src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.7)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.7)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.7)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.7)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.7)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.7)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.7)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.7)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.7)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.7)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.7)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.7)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.7)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.7)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.7)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning_app/core/constants.py, src/lightning_app/runners/cloud.py, src/lightning_app/source_code/copytree.py, src/lightning_app/source_code/local.py, src/lightning_app/utilities/clusters.py, requirements/app/base.txt.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

requirements/app/base.txt

awaelchli

I think this addresses #15012 too. Would you agree?

src/lightning_app/runners/cloud.py

src/lightning_app/source_code/local.py

src/lightning_app/runners/cloud.py

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]>

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> (cherry picked from commit 04886ed)

ethanwharris added 2 commits January 20, 2023 20:51

[App] Refactor cloud dispatch and update to new API

d0a077e

Clean

c8ecb9e

ethanwharris requested review from tchaton, lantiga, awaelchli, hhsecond and Borda as code owners January 20, 2023 20:56

github-actions bot added the app (removed) Generic label for Lightning App package label Jan 20, 2023

ethanwharris added 4 commits January 20, 2023 21:22

Docstrings + typing

c9c70ba

Fix hanging test

4f06d81

Mypy

9acade9

Fix CLI start test

aec1131

ethanwharris changed the title ~~[WIP] [App] Refactor cloud dispatch and update to new API~~ [App] Refactor cloud dispatch and update to new API Jan 20, 2023

Fix cli exception test

e4bcc72

ethanwharris self-assigned this Jan 20, 2023

ethanwharris added 2 commits January 20, 2023 22:05

Mock cloud backend to try to stop hanging

c4734f0

Fixes

bc7bf2e

carmocca approved these changes Jan 21, 2023

View reviewed changes

requirements/app/base.txt Outdated Show resolved Hide resolved

awaelchli approved these changes Jan 22, 2023

View reviewed changes

src/lightning_app/runners/cloud.py Outdated Show resolved Hide resolved

src/lightning_app/source_code/local.py Show resolved Hide resolved

src/lightning_app/runners/cloud.py Outdated Show resolved Hide resolved

mergify bot added the ready PRs ready to be merged label Jan 22, 2023

ethanwharris and others added 4 commits January 23, 2023 10:23

Apply suggestions from code review

9ddc003

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]>

Move public methods to top

e5379f7

Try something

64ac2e1

Connect to / lookup app by name

7d79acc

ethanwharris enabled auto-merge (squash) January 23, 2023 11:43

ethanwharris merged commit 04886ed into master Jan 23, 2023

ethanwharris deleted the feature/cloudspace_dispatch branch January 23, 2023 12:41

Borda pushed a commit that referenced this pull request Feb 9, 2023

[App] Refactor cloud dispatch and update to new API (#16456)

e1e30b8

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> (cherry picked from commit 04886ed)

awaelchli added this to the v1.9.x milestone Feb 10, 2023

lantiga pushed a commit that referenced this pull request Feb 10, 2023

[App] Refactor cloud dispatch and update to new API (#16456)

0aea257

Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> (cherry picked from commit 04886ed)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[App] Refactor cloud dispatch and update to new API #16456

[App] Refactor cloud dispatch and update to new API #16456

ethanwharris commented Jan 20, 2023 •

edited

Loading

github-actions bot commented Jan 20, 2023 •

edited

Loading

awaelchli left a comment

[App] Refactor cloud dispatch and update to new API #16456

[App] Refactor cloud dispatch and update to new API #16456

Conversation

ethanwharris commented Jan 20, 2023 • edited Loading

What does this PR do?

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

PR review

Did you have fun?

github-actions bot commented Jan 20, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

awaelchli left a comment

Choose a reason for hiding this comment

ethanwharris commented Jan 20, 2023 •

edited

Loading

github-actions bot commented Jan 20, 2023 •

edited

Loading