Releasing/1.8.4 extra2 (#15960)

* [docs] Include all components in the API reference (#15805) * Update docs Co-authored-by: Jirka Borovec <[email protected]> (cherry picked from commit e6f4c84) * Bump playwright from 1.27.1 to 1.28.0 in /requirements (#15903) * Bump playwright from 1.27.1 to 1.28.0 in /requirements Bumps [playwright](https://github.com/Microsoft/playwright-python) from 1.27.1 to 1.28.0. - [Release notes](https://github.com/Microsoft/playwright-python/releases) - [Commits](microsoft/playwright-python@v1.27.1...v1.28.0) --- updated-dependencies: - dependency-name: playwright dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * 1.28 Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jirka <[email protected]> (cherry picked from commit 73a6dbe) * [App] Add `configure_layout` method for works (#15926) * Add `configure_layout` method for works * Check for api access availability * Updates from review * Update CHANGELOG.md * Apply suggestions from code review Co-authored-by: Sherin Thomas <[email protected]> (cherry picked from commit d5b9c67) * Don't try to aggregate `requirements/__pycache__/base.txt` in setuptools (#15775) Exlucde __pycache__ in setuptools (cherry picked from commit ca5ca0e) * [App] Multiprocessing-safe work pickling (#15836) (cherry picked from commit df67833) * Multinode on MPS (#15748) * Fix restarting attribute for lr finder * update lite executor * update trainer executor * update spawn executor * add multinode component tests * add testing helpers * add lite tests * add trainer tests * update changelog * update trainer * update workflow * update tests * debug * add reason for skipif * Apply suggestions from code review * switch skipif Co-authored-by: Jirka <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Carlos Mocholí <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]> Co-authored-by: Jirka Borovec <[email protected]> (cherry picked from commit 36aecde) * [App] Resolve PythonServer on M1 (#15949) Co-authored-by: thomas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 904323b) * Lite: Fix DataLoader shuffling when using DistributedSampler (#15931) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 3004f13) * [App] Temporarily disable ready (#15958) (cherry picked from commit d0b101c) * update date * Fix restarting attribute for lr finder (#15620) (cherry picked from commit 15184c6) * [App] Improve pdb for multiprocessing (#15950) Co-authored-by: thomas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 482b279) * [App] Improve debug triggering (#15951) (cherry picked from commit 772d121) * [App] Add automatic conversion to structures (#15961) (cherry picked from commit 67a47d4) * Make LightningModule torch.jit.script-able again (#15947) * Make LightningModule torch.jit.script-able again * remove skip Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit b5fa896) * Fix ImportErrors on Multinode if package not present (#15963) (cherry picked from commit cbd4dd6) * Fix typo in definition of world size in docs (#15954) (cherry picked from commit 7a1e0e8) * [App] Enable running an app from the Gallery (#15941) Co-authored-by: thomas <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ethan Harris <[email protected]> Co-authored-by: Jirka <[email protected]> (cherry picked from commit 4983083) Co-authored-by: Akihiro Nitta <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ethan Harris <[email protected]> Co-authored-by: Sherin Thomas <[email protected]> Co-authored-by: Justus Schock <[email protected]> Co-authored-by: thomas chaton <[email protected]> Co-authored-by: Adrian Wälchli <[email protected]>
Lightning-AI · Dec 8, 2022 · 7eb5ff5 · 7eb5ff5
1 parent e0e75d1
commit 7eb5ff5
Show file tree

Hide file tree

Showing 50 changed files with 1,269 additions and 358 deletions.
diff --git a/.actions/setup_tools.py b/.actions/setup_tools.py
@@ -179,7 +179,7 @@ def _load_aggregate_requirements(req_dir: str = "requirements", freeze_requireme
         load_requirements(d, file_name="base.txt", unfreeze=not freeze_requirements)
         for d in glob.glob(os.path.join(req_dir, "*"))
         # skip empty folder as git artefacts, and resolving Will's special issue
-        if os.path.isdir(d) and len(glob.glob(os.path.join(d, "*"))) > 0
+        if os.path.isdir(d) and len(glob.glob(os.path.join(d, "*"))) > 0 and "__pycache__" not in d
     ]
     if not requires:
         return None

diff --git a/.azure/app-cloud-e2e.yml b/.azure/app-cloud-e2e.yml
@@ -51,7 +51,7 @@ jobs:
   - job: App_cloud_e2e_testing
     pool: azure-cpus
     container:
-      image: mcr.microsoft.com/playwright/python:v1.27.1-focal
+      image: mcr.microsoft.com/playwright/python:v1.28.0-focal
       options: "--shm-size=4gb"
     strategy:
       matrix:

diff --git a/.github/workflows/ci-app-tests.yml b/.github/workflows/ci-app-tests.yml
@@ -99,7 +99,7 @@ jobs:
 
       - name: Adjust tests
         if: ${{ matrix.pkg-name == 'lightning' }}
-        run: python .actions/assistant.py copy_replace_imports --source_dir="./tests" --source_import="lightning_app" --target_import="lightning.app"
+        run: python .actions/assistant.py copy_replace_imports --source_dir="./tests" --source_import="lightning_app,lightning_lite,pytorch_lightning" --target_import="lightning.app,lightning.lite,lightning.pytorch"
 
       - name: Adjust examples
         if: ${{ matrix.pkg-name != 'lightning' }}

diff --git a/docs/source-app/api_references.rst b/docs/source-app/api_references.rst
@@ -32,11 +32,19 @@ ___________________
     :nosignatures:
     :template: classtemplate_no_index.rst
 
+    ~database.client.DatabaseClient
+    ~database.server.Database
     ~python.popen.PopenPythonScript
     ~python.tracer.TracerPythonScript
     ~training.LightningTrainerScript
     ~serve.gradio.ServeGradio
     ~serve.serve.ModelInferenceAPI
+    ~serve.python_server.PythonServer
+    ~serve.streamlit.ServeStreamlit
+    ~multi_node.base.MultiNode
+    ~multi_node.lite.LiteMultiNode
+    ~multi_node.pytorch_spawn.PyTorchSpawnMultiNode
+    ~multi_node.trainer.LightningTrainerMultiNode
     ~auto_scaler.AutoScaler
 
 ----

diff --git a/docs/source-pytorch/clouds/cluster_intermediate_1.rst b/docs/source-pytorch/clouds/cluster_intermediate_1.rst
@@ -24,7 +24,7 @@ PyTorch Lightning follows the design of `PyTorch distributed communication packa
 
 - *MASTER_PORT* - required; has to be a free port on machine with NODE_RANK 0
 - *MASTER_ADDR* - required (except for NODE_RANK 0); address of NODE_RANK 0 node
-- *WORLD_SIZE* - required; how many nodes are in the cluster
+- *WORLD_SIZE* - required; the total number of GPUs/processes that you will use
 - *NODE_RANK* - required; id of the node in the cluster
 
 .. _training_script_setup:

diff --git a/examples/app_dag/requirements.txt b/examples/app_dag/requirements.txt
@@ -1,2 +1,2 @@
-sklearn
+scikit-learn
 pandas
diff --git a/pyproject.toml b/pyproject.toml
@@ -30,14 +30,12 @@ files = [
     "src/lightning_lite",
     "src/lightning_app",
 ]
+# This section is for folders with "-" as they are not valid python modules
 exclude = [
+    "src/lightning_app/cli/app-template",
     "src/lightning_app/cli/component-template",
     "src/lightning_app/cli/pl-app-template",
     "src/lightning_app/cli/react-ui-template",
-    "src/lightning_app/cli/app-template",
-    "src/lightning_app/components/database",
-    "src/lightning_app/components/multi_node",
-    "src/lightning_app/frontend/just_py/just_py",
 ]
 install_types = "True"
 non_interactive = "True"
@@ -67,7 +65,9 @@ module = [
     "lightning_app.api.request_types",
     "lightning_app.cli.commands.app_commands",
     "lightning_app.cli.commands.connection",
-    "lightning_app.cli.react-ui-template.example_app",
+    "lightning_app.cli.commands.lightning_cli",
+    "lightning_app.cli.commands.cmd_install",
+    "lightning_app.cli.cmd_install",
     "lightning_app.components.database.client",
     "lightning_app.components.database.server",
     "lightning_app.components.database.utilities",
@@ -94,6 +94,7 @@ module = [
     "lightning_app.frontend.streamlit_base",
     "lightning_app.frontend.utils",
     "lightning_app.frontend.web",
+    "lightning_app.perf.pdb",
     "lightning_app.runners.backends.__init__",
     "lightning_app.runners.backends.backend",
     "lightning_app.runners.backends.cloud",

diff --git a/requirements/app/test.txt b/requirements/app/test.txt
@@ -4,7 +4,7 @@ pytest==7.2.0
 pytest-timeout==2.1.0
 pytest-cov==4.0.0
 pytest-doctestplus>=0.9.0
-playwright==1.27.1
+playwright==1.28.0
 httpx
 trio<0.22.0
 pympler

diff --git a/requirements/app/ui.txt b/requirements/app/ui.txt
@@ -1,2 +1,2 @@
-streamlit>=1.3.1, <=1.11.1
+streamlit>=1.0.0, <=1.15.2
 panel>=0.12.7, <=0.13.1
diff --git a/src/lightning_app/CHANGELOG.md b/src/lightning_app/CHANGELOG.md
@@ -4,18 +4,21 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
-## [1.8.4] - 2022-12-06
+## [1.8.4] - 2022-12-08
 
 ### Added
 
 - Add `code_dir` argument to tracer run ([#15771](https://github.com/Lightning-AI/lightning/pull/15771))
 - Added the CLI command `lightning run model` to launch a `LightningLite` accelerated script ([#15506](https://github.com/Lightning-AI/lightning/pull/15506))
 - Added the CLI command `lightning delete app` to delete a lightning app on the cloud ([#15783](https://github.com/Lightning-AI/lightning/pull/15783))
 - Added a CloudMultiProcessBackend which enables running a child App from within the Flow in the cloud ([#15800](https://github.com/Lightning-AI/lightning/pull/15800))
+- Utility for pickling work object safely even from a child process ([#15836](https://github.com/Lightning-AI/lightning/pull/15836))
 - Added `AutoScaler` component ([#15769](https://github.com/Lightning-AI/lightning/pull/15769))
 - Added the property `ready` of the LightningFlow to inform when the `Open App` should be visible ([#15921](https://github.com/Lightning-AI/lightning/pull/15921))
 - Added private work attributed `_start_method` to customize how to start the works ([#15923](https://github.com/Lightning-AI/lightning/pull/15923))
-
+- Added a `configure_layout` method to the `LightningWork` which can be used to control how the work is handled in the layout of a parent flow ([#15926](https://github.com/Lightning-AI/lightning/pull/15926))
+- Added the ability to run a Lightning App or Component directly from the Gallery using `lightning run app organization/name` ([#15941](https://github.com/Lightning-AI/lightning/pull/15941))
+- Added automatic conversion of list and dict of works and flows to structures ([#15961](https://github.com/Lightning-AI/lightning/pull/15961))
 
 ### Changed
 
@@ -37,13 +40,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed SSH CLI command listing stopped components ([#15810](https://github.com/Lightning-AI/lightning/pull/15810))
 - Fixed bug when launching apps on multiple clusters ([#15484](https://github.com/Lightning-AI/lightning/pull/15484))
 - Fixed Sigterm Handler causing thread lock which caused KeyboardInterrupt to hang ([#15881](https://github.com/Lightning-AI/lightning/pull/15881))
+- Fixed MPS error for multinode component (defaults to cpu on mps devices now as distributed operations are not supported by pytorch on mps) ([#15748](https://github.com/Ligtning-AI/lightning/pull/15748))
 - Fixed the work not stopped when successful when passed directly to the LightningApp ([#15801](https://github.com/Lightning-AI/lightning/pull/15801))
 - Fixed the PyTorch Inference locally on GPU ([#15813](https://github.com/Lightning-AI/lightning/pull/15813))
 - Fixed the `enable_spawn` method of the `WorkRunExecutor` ([#15812](https://github.com/Lightning-AI/lightning/pull/15812))
 - Fixed require/import decorator ([#15849](https://github.com/Lightning-AI/lightning/pull/15849))
-
 - Fixed a bug where using `L.app.structures` would cause multiple apps to be opened and fail with an error in the cloud ([#15911](https://github.com/Lightning-AI/lightning/pull/15911))
-
+- Fixed PythonServer generating noise on M1 ([#15949](https://github.com/Lightning-AI/lightning/pull/15949))
+- Fixed multiprocessing breakpoint ([#15950](https://github.com/Lightning-AI/lightning/pull/15950))
+- Fixed detection of a Lightning App running in debug mode ([#15951](https://github.com/Lightning-AI/lightning/pull/15951))
+- Fixed `ImportError` on Multinode if package not present ([#15963](https://github.com/Lightning-AI/lightning/pull/15963))
 
 ## [1.8.3] - 2022-11-22