Add cli commands

[Leahh02]

This PR adds cli commands for the curl requests for the ci demonstrator. I implemented it very similarly to the way the requests are implemented in the ci demonstrator, but I can change it to use the requests library, for instance.

[kchilleri]

Tested these commands through Gitlab CI pipeline and they all work as expected. I'm going to think of possible unit tests now and look at the actual code.

@pagrubel do we want this remote stuff to be a part of our documentation?

[Leahh02]

I added some simple documentation to rest_api.rst, we will discuss the location and content.

I realized I didn't add any error handling into remote_client.py, I will consider adding this.

[Leahh02]

I learned about modules to test fastapi apps: https://fastapi.tiangolo.com/tutorial/testing/ and a module to test typer apps: https://typer.tiangolo.com/tutorial/testing/

[kchilleri]

I'm wondering why we are upgrading to this version?

I think we should use a range of versions (e.g. fastapi = "^0.109.2") instead of a specific version. This will avoid overly constraining dependencies and conflicts.

[kchilleri]

I guess this is more of a team discussion, though.

[kchilleri]

^ this will be taken care of in a separate PR.

[kchilleri]

@Leahh02 is there any way you can add a remote command for beeflow core status?

Thankfully, this has already been added to remote.py, in def get_core_status(), and you'd only have to replace this curl request:

curl ssh_target:port/core/status/

I think this would be very useful to have when interacting remotely because we'll be able to check that all the appropriate components are up and running.

[Leahh02]

@Leahh02 is there any way you can add a remote command for beeflow core status?

Thankfully, this has already been added to remote.py, in def get_core_status(), and you'd only have to replace this curl request:

curl ssh_target:port/core/status/

I think this would be very useful to have when interacting remotely because we'll be able to check that all the appropriate components are up and running.

I can absolutely do that!

[Leahh02]

@Leahh02 is there any way you can add a remote command for beeflow core status?
Thankfully, this has already been added to remote.py, in def get_core_status(), and you'd only have to replace this curl request:

curl ssh_target:port/core/status/

I think this would be very useful to have when interacting remotely because we'll be able to check that all the appropriate components are up and running.

I can absolutely do that!

My most recent commit adds this in along with a test. This was a quick fix, it just took me a while to do it because I had to finish other stuff up first :)

[kchilleri]

These remote commands have been tested both locally and within a Gitlab CI pipeline. Everything seems to work well. Great job!

[arhall0]

This should probably just be done with the pythonjson package. Otherwise this has OS issues.

[arhall0]

This currently doesn't work for copying from local to remote.

[kchilleri]

Okay, let me test it again locally (outside of Gitlab CI pipeline)

[Leahh02]

It should be fixed now to work locally. I don't know how to test it to see if it still works with the Gitlab CI pipeline though. I'll update the unit tests now

[arhall0]

I set up BEE on remote by making sure my config had remote_api = True and that I had a unique remote_api_port. Then I did beeflow core start

On my local machine I made sure to set remote_api_port to the same port as remote. beeflow core start is unnecessary for local.

I checked each command from local.

• beeflow remote connection, beeflow remote droppoint, beeflow remote core-status all worked without issue
• beeflow remote submit worked locally after manually setting up the files on remote. From ~/.beeflow/droppoint:
  -- beeflow package $BEE_PATH/examples/cat-grep-tar .
  -- mkdir cat-grep-tar-workdir
  -- cp $BEE_PATH/examples/cat-grep-tar/lorem.txt cat-grep-tar-workdir/

[arhall0]

This command is making the directory locally. But also, the directory should exist on remote if remote was setup properly.

This errors for me because I can't make the folder locally because of permissions issues. If I comment this out, it works as expected.

[arhall0]

After the most recent commits I can run a workflow end-to-end from local on remote. For cat-grep-tar these are the commands:

beeflow remote droppoint $SSH_TARGET
beeflow package $BEE_PATH/examples/cat-grep-tar .
beeflow remote copy $USER $SSH_TARGET cat-grep-tar.tgz
mkdir cat-grep-tar-workdir
cp $BEE_PATH/examples/cat-grep-tar/lorem.txt cat-grep-tar-workdir
beeflow remote copy $USER $SSH_TARGET cat-grep-tar-workdir
beeflow remote submit $SSH_TARGET test_remote cat-grep-tar.tgz workflow.cwl input.yml

I think that this works well. It's maybe a bit verbose so maybe there are things we could do in the future to improve the workflow.

Another improvement for the future would be a way to check the status of your job & pull down the results to local. Automating these things to as few as commands as possible would improve their utility over just using things like rsync and scp.

[arhall0]

In general test coverage is good. For remote_client.py, test coverage is missing mostly for lines which are checking for errors or exceptions:

--------- coverage: platform darwin, python 3.10.13-final-0 ----------
Name                                                  Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------
beeflow/client/remote_client.py                          80     60    25%   22, 27, 36-48, 54-71, 79-102, 112-141, 147-159

This is probably not a big deal.

I also noticed the droppoint folder is created when you run tests. This also probably doesn't matter too much.

[kchilleri]

I still need to test these new changes on the Gitlab CI work, please don't merge yet!