Run Command from App Comments

[rlizzo]

What does this PR do?

We want a no-brainer experience while making apps.

• Requirements definitely flex too much brainpower
• We are enabling ! syntax on top of entry files for those wishing to prototype and move super fast.

When running an app, we allow for bash commands to be run in the user (or clouds) current python environment in order to modify the environment & install requirements.

Any top level comment lines in the app or work entrypoint file will be executed if they contain the ! symbol (just like in jupyter notebooks). For example:

# v0_app.py            # <--- not a command
# !conda info -e       # <--- a command
# !echo "hello world"  # <--- a command
# ! pip install lmdb   # <--- a command
import lightning        # <--- not a command, end parsing here
# !echo "after parse"   # <--- not a command, occurred after parsing end

if we run lightning run app ./app.py now, we will see the following output:

(lightning) rick@rick-pop:~/projects/lightning$ lightning run app examples/app_v0/app.py 
Your Lightning App is starting. This won't take long.
# conda environments:
#
base                     /home/rick/miniconda3
grid                     /home/rick/miniconda3/envs/grid
lightning             *  /home/rick/miniconda3/envs/lightning

hello world
Collecting lmdb
  Downloading lmdb-1.3.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (305 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 305.9/305.9 kB 4.9 MB/s eta 0:00:00
Installing collected packages: lmdb
Successfully installed lmdb-1.3.0

Please note that the environment the user above is in is preserved. In this example I was using python within a conda environment, the command was able to pick up that I ran this from a conda environment and it installed the lmdb package in that conda environment.

If the user did not have conda installed, it would have errorer'd out with the standard #!/bin/sh 1: conda: not found command inline and informed me what file, line number, and command caused the error (along with what traceback to look at to address the problem)

Before the e2e test will pass, we need to make a backend release which I am preparing PRs for now.

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

Yes I had fun coding 😄

cc @Borda

[tchaton]

LGTM !

[ethanwharris]

LGTM