add coreos-assembler remote-session command

[dustymabe]

This adds a remote-session subcommand to coreos-assembler. The intent is for us to be able to create remote sessions on remote servers via podman --remote and execute builds there. This is especially attractive when considering that it elegantly enables us to do builds for a different architecture by targeting a remote host.

Convenience subcommands are provided to enable easily execute arbitary remote commands (via cosa remote-session exec shell) and also copying files back and forth via an rsync-like interface with cosa remote-session sync.

Here's what a typical session may look like:

// initialize local dir
mkdir fcos && cd fcos
cosa init https://github.com/coreos/fedora-coreos-config.git
cosa buildfetch

// initialize remote session and sync files
REMOTEHOST=myhost.example.com; REMOTEUSER=builder; REMOTEUID=1001
export CONTAINER_HOST="ssh://${REMOTEUSER}@${REMOTEHOST}/run/user/${REMOTEUID}/podman/podman.sock"
export COREOS_ASSEMBLER_REMOTE_SESSION=$(cosa remote-session create)
cosa remote-session sync ./ :/srv/

// execute build/tests
cosa fetch && cosa build && cosa kola run

// sync back results
cosa remote-session sync :/srv/ ./

// destroy remote session
cosa remote-session destroy
unset COREOS_ASSEMBLER_REMOTE_SESSION

[cgwalters]

Overall looks sane to me

[cgwalters]

One thing I'm hazy on is how this works with the "context directory"; e.g. is the cosa init and the git clone expected to happen remotely? Or is it local, and then podman syncs it?

[dustymabe]

it should be able to work either way. You can cosa init locally and sync it over first or you can set up the remote session and cosa init on the remote.

[cgwalters]

Sounds cool and magical; not a blocker for merge but would be nice add this new flow in the docs!

Or, another approach may be to land the fedora-coreos-pipeline changes, and then at least just link to that code. (I tend to learn a lot by reading about how APIs/CLIs are used in the real world and not just docs)

[dustymabe]

I'm working on the pipeline changes and I'll post them up shortly (testing them now). Trying to work out all the kinks first.

[dustymabe]

#2979 (comment)

[dustymabe]

I think I've done sufficient testing on this now. I have a ton of changes to the pipeline that I need to separate into individual commits, which is going to take some time. For now here is a view of what this looks like:

https://github.com/dustymabe/fedora-coreos-pipeline/blob/dusty-pipeline-refactor-persistent/jobs/build-arch.Jenkinsfile#L198-L207

I think this can merge in the meantime.

[jlebon]

Nice work!

[cgwalters]

Followup thought from a demo related to this:

I think the pipeline at least should have cosa remote-session create be "owner" the container's lifecycle, instead of a timeout 4h.

Take the example of a pipeline build that is aborted due to a test error. AIUI in the current flow, the remote container will "leak" until it times out.

In this suggestion, there is an "active binding" between a process in the container on the pipeline and the remote container. When the pipeline container dies, it would break a TCP connection with the remote container and cause it to also die.

[dustymabe]

I think the pipeline at least should have cosa remote-session create be "owner" the container's lifecycle, instead of a timeout 4h.

This is easy to do by just putting the cosa remote-session destroy in a finally block. If somehow it doesn't get run for whatever reason (Jenkins barfed or got shut down) the container still will kill itself.