Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for custom cloud compute configurations for Flows #14831

Merged
merged 40 commits into from
Oct 25, 2022
Merged

Add support for custom cloud compute configurations for Flows #14831

merged 40 commits into from
Oct 25, 2022

Conversation

nicolai86
Copy link
Contributor

@nicolai86 nicolai86 commented Sep 21, 2022 •
edited
Loading

What does this PR do?

This PR exposes the custom cloud compute configuration we support on the lightning AI platform as part of the lightning framework.
It depends on an update lightning-cloud package, which is why this PR is pending right now.

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

  • Was this discussed/approved via a GitHub issue? (not for typos and docs)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests? (not for typos and docs)
  • Did you verify new and existing tests pass locally with your changes?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

  • Is this pull request ready for review? (if not, please submit in draft mode)
  • Check that all items from Before submitting are resolved
  • Make sure the title is self-explanatory and the description concisely explains the PR
  • Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

cc @Borda

@github-actions github-actions bot added the app (removed) Generic label for Lightning App package label Sep 21, 2022
@nicolai86 nicolai86 added app (removed) Generic label for Lightning App package and removed app (removed) Generic label for Lightning App package labels Sep 21, 2022
@nicolai86 nicolai86 added this to the app:0.6.x milestone Sep 21, 2022
@nicolai86 nicolai86 marked this pull request as ready for review September 27, 2022 19:12
@awaelchli awaelchli added the feature Is an improvement or enhancement label Sep 27, 2022
tchaton
tchaton reviewed Sep 28, 2022
View reviewed changes
src/lightning_app/core/app.py Show resolved Hide resolved
src/lightning_app/runners/cloud.py Outdated Show resolved Hide resolved
hhsecond
hhsecond reviewed Sep 28, 2022
View reviewed changes
src/lightning_app/core/app.py Outdated Show resolved Hide resolved
src/lightning_app/core/app.py Outdated Show resolved Hide resolved
@mergify mergify bot added ready PRs ready to be merged has conflicts and removed ready PRs ready to be merged labels Oct 20, 2022
@mergify mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Oct 20, 2022
@awaelchli
Copy link
Contributor

awaelchli commented Oct 21, 2022 •
edited
Loading

@nicolai86 I fixed the docs build for you (hopefully).
@edenlightning could you also take a look at the docs here?

@mergify mergify bot added has conflicts and removed ready PRs ready to be merged labels Oct 21, 2022
@mergify mergify bot added ready PRs ready to be merged and removed has conflicts ready PRs ready to be merged labels Oct 24, 2022
tchaton
tchaton approved these changes Oct 25, 2022
View reviewed changes
Copy link
Contributor

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@nicolai86 nicolai86 merged commit 13baad5 into master Oct 25, 2022
@nicolai86 nicolai86 deleted the nicolai86/ENG-445 branch October 25, 2022 18:29
@carmocca carmocca modified the milestones: v1.9, v1.8 Oct 25, 2022
carmocca added a commit that referenced this pull request Oct 25, 2022
@nicolai86 @tchaton
@ziyadsheeba @otaj @carmocca @jerome-habana @awaelchli @pre-commit-ci @akihironitta @adamjstewart @donlapark
Add support for custom cloud compute configurations for Flows (#14831)
7aa1620 
* use more recent lightning cloud launcher

* allow LightningApp to use custom cloud compute for flows

* feedback from adrian

* adjust other cloud tests

* update

* update

* update commens

* Update src/lightning_app/core/app.py

Co-authored-by: Sherin Thomas <[email protected]>

* Close profiler when `StopIteration` is raised (#14945)

* Find last checkpoints on restart (#14907)


Co-authored-by: Carlos Mocholí <[email protected]>

* Remove unused gcsfs dependency (#14962)

* Update hpu mixed precision link (#14974)

Signed-off-by: Jerome <[email protected]>

* Bump version of fsspec (#14975)

fsspec verbump

* Fix TPU test CI (#14926)

* Fix TPU test CI

* +x first

* Lite first to uncovert errors faster

* Fixes

* One more

* Simplify XLALauncher wrapping to avoid pickle error

* debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug commit successful. Trying local definitions

* Require tpu for mock test

* ValueError: The number of devices must be either 1 or 8, got 4 instead

* Fix mock test

* Simplify call, rely on defaults

* Skip OSError for now. Maybe upgrading will help

* Simplify launch tests, move some to lite

* Stricter typing

* RuntimeError: Accessing the XLA device before processes have spawned is not allowed.

* Revert "RuntimeError: Accessing the XLA device before processes have spawned is not allowed."

This reverts commit f65107e.

* Alternative boring solution to the reverted commit

* Fix failing test on CUDA machine

* Workarounds

* Try latest mkl

* Revert "Try latest mkl"

This reverts commit d06813a.

* Wrong exception

* xfail

* Mypy

* Comment change

* Spawn launch refactor

* Accept that we cannot lazy init now

* Fix mypy and launch test failures

* The base dockerfile already includes mkl-2022.1.0 - what if we use it?

* try a different mkl version

* Revert mkl version changes

Co-authored-by: awaelchli <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Akihiro Nitta <[email protected]>

* Trainer: fix support for non-distributed PyTorch (#14971)

* Trainer: fix non-distributed use
* Update CHANGELOG

* fixes typing errors in rich_progress.py (#14963)

* revert default cloud compute rename

* allow LightningApp to use custom cloud compute for flows

* feedback from adrian

* update

* resolve merge with master conflict

* remove preemptible

* update CHANGELOG

* add basic flow cloud compute documentation

* fix docs build

* add missing symlink

* try to fix sphinx

* another attempt for docs

* fix new test

Signed-off-by: Jerome <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
Co-authored-by: Sherin Thomas <[email protected]>
Co-authored-by: Ziyad Sheebaelhamd <[email protected]>
Co-authored-by: otaj <[email protected]>
Co-authored-by: Carlos Mocholí <[email protected]>
Co-authored-by: Jerome Anand <[email protected]>
Co-authored-by: awaelchli <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Akihiro Nitta <[email protected]>
Co-authored-by: Adam J. Stewart <[email protected]>
Co-authored-by: DP <[email protected]>
@dmitsf dmitsf mentioned this pull request Oct 27, 2022
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rohitgr7 rohitgr7 rohitgr7 left review comments

@hhsecond hhsecond hhsecond approved these changes

@tchaton tchaton tchaton approved these changes

@awaelchli awaelchli awaelchli approved these changes

@manskx manskx Awaiting requested review from manskx

@lantiga lantiga Awaiting requested review from lantiga lantiga is a code owner

@williamFalcon williamFalcon Awaiting requested review from williamFalcon

@Borda Borda Awaiting requested review from Borda Borda is a code owner

@kaushikb11 kaushikb11 Awaiting requested review from kaushikb11

@carmocca carmocca Awaiting requested review from carmocca

@justusschock justusschock Awaiting requested review from justusschock justusschock is a code owner

@otaj otaj Awaiting requested review from otaj

@edenlightning edenlightning Awaiting requested review from edenlightning

@akihironitta akihironitta Awaiting requested review from akihironitta

@SkafteNicki SkafteNicki Awaiting requested review from SkafteNicki

Assignees

@nicolai86 nicolai86

Labels
app (removed) Generic label for Lightning App package feature Is an improvement or enhancement ready PRs ready to be merged
Projects
None yet
Milestone
v1.8
Development

Successfully merging this pull request may close these issues.