feat(emulator): allowing to download and run emulator from URL

[grdddj]

Connected with trezor/trezor-firmware#1932 and #49

Ability to specify emulator URL which will be downloaded and run by tenv.

Tested using emulators from Gitlab CI -
https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/1839236800/artifacts/raw/core/build/unix/trezor-emu-core
https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/1839236825/artifacts/raw/legacy/firmware/trezor.elf

[tsusanka]

Awesome 🎉.

There is one thing I do not know: do we want to support generic URLs or are we fine with Gitlab links? In other words do we want to support downloads from anywhere?

Because if we limit it for Gitlab, we can simplify some code and also the UI. We would fill the artifacts names automatically. We could also request the job ID (1839236800) only.

Do we see use-cases where you would want to use some URL outside of Gitlab? cc @matejcik @mroz22

We could of course do both..

---

Also this is a great start and if we want to support branches easily we can do that by crafting a special URL. This URL https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/artifacts/master/download?job=core%20unix%20frozen%20debug%20build always leads to the latest core build on master. Replacing the master with any branch will do the same trick. But let's finish this PR first and worry about that then.

[matejcik]

The primary usecase for me would be to supply a local file for use.

This would cover the gitlab usecase too, we can fetch the artifact however we like and supply the local file to tenv.

Is there a usecase that actually wants tenv to download a specified URL?

[matejcik]

To be specific: although perhaps convenient in some situations, I never need to upload a new emulator while tenv is running. A well documented docker cp would work just fine, together with version "locally supplied" parallel to master.
-- that, or the ability to specify local path at runtime (but the URL thing would work with a file path too, maybe?)

[grdddj]

Is there a usecase that actually wants tenv to download a specified URL?

This would allow the firmware CI (where the artifacts get created) and any tests using tenv (Suite, Connect) being independent (running on different pipelines, environments, etc.). Also for the manual testing it could be useful so that QA or anybody else can test the behavior of any emulator being built at Gitlab or anywhere else.

[matejcik]

This would allow the firmware CI (where the artifacts get created) and any tests using tenv (Suite, Connect) being independent (running on different pipelines, environments, etc.).

supplying a local file would work just as well -- the CI pipeline instances are not talking to the same docker instance running tenv, are they?

Also for the manual testing it could be useful so that QA or anybody else

Yes, this seems nice. In that case we can do what @tsusanka suggests and accept a link to the job instead of to the artifact.
I'm not sure how much it helps though. It's a UX nicety but other than that the code is essentially the same?

[grdddj]

To be specific: although perhaps convenient in some situations, I never need to upload a new emulator while tenv is running. A well documented docker cp would work just fine, together with version "locally supplied" parallel to master. -- that, or the ability to specify local path at runtime (but the URL thing would work with a file path too, maybe?)

I did some prototype with docker cp some time ago in trezor/trezor-suite#4107 - after starting the tenv container we can copy any emulators into it and use them as 2-master - alternatively giving them some special name like 2-custom. We can theoretically do it in runtime as well, however probably only by pushing the emulator inside again by docker cp, I do not think docker container could get files from the local machine freely (but I would be glad to be mistaken)

[matejcik]

I see, I must have missed that. It looks like it would work fine for our CI usage. Specific name like custom would be a better solution than X-master (probably not even including version, because tenv can't know and verify that it's the right version), but that's a cosmetic thing.

[grdddj]

the CI pipeline instances are not talking to the same docker instance running tenv, are they?

Each pipeline using tenv is getting the image from registry.gitlab.com/satoshilabs/trezor/trezor-user-env/trezor-user-env:latest, so I think they do not use the same instance, just the same base image.

[grdddj]

It looks like it would work fine for our CI usage.

So we could wait after all the jobs in build CI stage are finished, take the core/build/unix/trezor-emu-core file, transfer it into the tenv as 2-custom and tell the Connect tests to use 2-custom tests?

Connect's main testing script, https://github.com/trezor/connect/blob/develop/tests/run.sh, already has the -f option to specify the emulator version, so we can supply that 2-custom there.

I will be trying to implement this pipeline as our CI stage in trezor/trezor-firmware#1972

We should probably test is against T1 emu as well, but it has lower priority (as it is not changing so much)

[matejcik]

not all, just the one that produces the file we need (so presumably "core unix debug")
otherwise, yes, exactly.

[tsusanka]

Sooo if I sum it up: we will do FW tests using simple cp as we'll require the build job using needs so the build will be present inside the container. This is nice and simple so 👍. @grdddj tackles this in trezor/trezor-firmware#1972 so let's continue the discussion there.

---

Now regarding this PR: could we still make use of this URL feature? Exactly as you say I can imagine some use cases for QA or Suite devs where you need to test some particular branch (like a Taproot one recently). It seems like a nice feature but it is true that we should not implement something until we properly understand the need. @mroz22 what do you think of this? Would you find it useful if you could set Gitlab URL in tenv and simply launch the emulator from such pipeline?

[tsusanka]

Let's change the URL to https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/1839236825/ and fill in the details

[grdddj]

Let's change the URL to https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/1839236825/ and fill in the details

Force-pushed after rebase on master and the above-mentioned change.

It is currently possible to input either Gitlab job link (https://gitlab.com/satoshilabs/trezor/trezor-firmware/-/jobs/1839236825/), just the job ID (1839236825) or the complete link, which makes sure other resources other that Gitlab CI could be used.

UX and UI is another unknown, but I believe the functionality is much more important here.

[tsusanka]

One tiny thing, feel free to force-push then :). Did not test, but let's see how it works :).

[tsusanka]

Let's remove the comment, I think the select box is enough.

[grdddj]

Removed in the force-pushed commit (force-pushed twice, second already rebased on current master, even though there was no conflict)

[grdddj]

One tiny thing, feel free to force-push then :). Did not test, but let's see how it works :).

Force-pushed after squashing everything

[tsusanka]

I am getting a conflict now. Probably due to #126.

[grdddj]

I am getting a conflict now. Probably due to #126.

Correct, there was a need of rebase. Force-pushed now