[App] Enable to register data connections

[tchaton]

What does this PR do?

This PR refactors connect to add sub ressources:

lightning connect data

lightning connect app

lightning connect X

NB: Need to merge some platforms stuffs for the tests to pass.

Fixes #<issue_number>

Before submitting

• Was this discussed/agreed via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @Borda

[github-actions]

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (slow, macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (slow, windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 pytorch_lightning: Azure HPU

Check ID	Status
pytorch-lightning (HPUs)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 pytorch_lightning: Azure IPU

Check ID	Status
pytorch-lightning (IPUs)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to pyproject.toml.

🟢 lightning_app: Tests workflow

Check ID	Status
app-pytest (macOS-11, lightning, 3.8, latest)	success	✅
app-pytest (macOS-11, lightning, 3.8, oldest)	success	✅
app-pytest (macOS-11, app, 3.9, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, latest)	success	✅
app-pytest (ubuntu-20.04, lightning, 3.8, oldest)	success	✅
app-pytest (ubuntu-20.04, app, 3.9, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, latest)	success	✅
app-pytest (windows-2022, lightning, 3.8, oldest)	success	✅
app-pytest (windows-2022, app, 3.8, latest)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, tests/tests_app/cli/test_connect.py, tests/tests_app/cli/test_connect_data.py, tests/tests_app/utilities/test_commands.py, requirements/app/base.txt.

🟢 lightning_app: Examples

Check ID	Status
app-examples (macOS-11, lightning, 3.9, latest)	success	✅
app-examples (macOS-11, lightning, 3.9, oldest)	success	✅
app-examples (macOS-11, app, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, latest)	success	✅
app-examples (ubuntu-20.04, lightning, 3.9, oldest)	success	✅
app-examples (ubuntu-20.04, app, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, latest)	success	✅
app-examples (windows-2022, lightning, 3.9, oldest)	success	✅
app-examples (windows-2022, app, 3.9, latest)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, tests/integrations_app/public/test_commands_and_api.py, requirements/app/base.txt.

🟢 lightning_app: Azure

Check ID	Status
App.cloud-e2e	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, requirements/app/base.txt, tests/integrations_app/public/test_commands_and_api.py.

🟢 lightning_app: Docs

Check ID	Status
make-doctest (app)	success	✅
make-html (app)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, requirements/app/base.txt, pyproject.toml.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to requirements/app/base.txt, src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, pyproject.toml.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/app/cli/commands/app_commands.py, src/lightning/app/cli/commands/cd.py, src/lightning/app/cli/commands/ls.py, src/lightning/app/cli/connect/__init__.py, src/lightning/app/cli/connect/app.py, src/lightning/app/cli/connect/data.py, src/lightning/app/cli/lightning_cli.py, requirements/app/base.txt.

🟢 link-check

Check ID	Status
markdown-link-check	success	✅

These checks are required after the changes to src/lightning/app/CHANGELOG.md.

---

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

[ethanwharris]

It feels a bit weird to me that lightning connect data and lightning connect app both exist in the same group as they are about two different kinds of connection. So lightning connect app doesn't feel right, it would be more like lightning connect to app - maybe lightning connect should be renamed entirely to lightning attach or something?

[codecov]

Codecov Report

Merging #16670 (4d833fc) into master (74ee699) will decrease coverage by 0%.
The diff coverage is 90%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #16670   +/-   ##
=======================================
- Coverage      81%      81%   -0%     
=======================================
  Files         445      436    -9     
  Lines       32500    32423   -77     
=======================================
- Hits        26431    26352   -79     
- Misses       6069     6071    +2     

[tchaton]

It feels a bit weird to me that lightning connect data and lightning connect app both exist in the same group as they are about two different kinds of connection. So lightning connect app doesn't feel right, it would be more like lightning connect to app - maybe lightning connect should be renamed entirely to lightning attach or something?

I agree. Let's re-align the CLI experience soon before Trainer 2.0. So we can improve on the wordings and consistency.

[awaelchli]

I agree with @ethanwharris