Use custom firmware

[szymonlesisz]

Currently we are working only with prefetched FW, it would be nice to have a possibility to work with custom build or custom trezor-firmware/branch or custom trezor-firmware-fork/branch

Usecase: testing connect against not merged branches/pull requests in trezor-firmware repo

[matejcik]

user story: I have a local checkout of the firmware repo with some things built. I want to run Suite e2e tests via trezor-user-env, against the locally built thing.

Options:

• i launch the emulator locally, tell the docker image to use it
• i launch the emulator and bridge locally, tell the docker image to use my bridge instead of launching its own
  • advantage: this way I can run Suite tests against a hardware device
• i submit the emulator binary to docker image, tell it to use it
  • drawback: this only works for frozen builds
• i tell the docker image to use my local checked-out directory
  • advantage: I can use non-frozen builds (which need to see the Python sources in the filesystem)
  • disadvantage: tenv probably doesn't know how to launch emulator in this configuration? (you need to be at the right working directory)

[grdddj]

We have agreed to try the option 3, where the binary file is already available (non-frozen build will need to be compiled, but it is a quick process).

Probably main question right now is: do we want to run the whole thing locally or using CI? (we can support even both)

• it can be theoretically rephrased as do we want to run only specific tests with new firmware, or all of them? (only specific -> locally is better, all of them -> CI is better)
  • Running the Suite e2e test locally can be a pain. I do not know about a good way how to run all the tests locally at once, without clicking each test individually in Cypress window. And even if it was possible, it would probably take a VERY long time, as in CI every stage runs in parralel, not sure if it can be done locally

Steps needed to be done in every case:

• having the binary emulator file (in case of TT, it is around 11 MB currently, so can be quite easily uploaded almost everywhere)
• tenv having access to it
• Suite e2e tests specifying this version

CI route

• binary emulator can be uploaded somewhere (into Gitlab)
• CI job could take the binary URL as an argument, store it in Suite e2e test code, which will be forwarding that to tenv with a request to spawn this emulator (sending that URL instead of 2-master)
• tenv will recognize this is not a normal version and will try to download it from that URL and then run it (it will need to download it only once, then tenv will know it already has it)

Local route

• quite similar as the CI route, just having the file locally and instead of giving argument to the CI job, we would give the binary location to the docker/docker-suite-test.sh in Suite - which could make sure the binary will be uploaded into the tenv container and that Suite tests will be requesting it instead of 2-master

[matejcik]

Probably main question right now is: do we want to run the whole thing locally or using CI? (we can support even both)

We want both.

CI usecase: running Suite e2e tests in the firmware pipeline -> Suite e2e job must pick the artifact built in previous stage.
This is probably higher priority.

Local usecase: local development, I might want to quickly iterate over a small number of Suite tests. I will then push the branch to CI where the CI job runs all of it.

[grdddj]

I will start with the local version and see how it goes.

Implementing that in CI seems much harder from my standpoint, as I am not very familiar with that (we would somehow need to mount the whole Suite CI stage onto firmware CI). It would also mean the firmware CI would be running longer, when we add more tests - but that is probably expected.

Suite e2e job must pick the artifact built in previous stage

Can the artifact be reached and downloaded by URL? It seems like the easiest solution, so that we only give this URL to Suite, it will forward it to tenv and it will download it and run it.

[grdddj]

Prototype of the local version was made into trezor/trezor-suite#4107

[tsusanka]

user story: I have a local checkout of the firmware repo with some things built. I want to run Suite e2e tests via trezor-user-env, against the locally built thing.

Do we really need to support locally built emulator? I am wondering how necessary that is. I can imagine it being useful but seems like smth we can live without. We have device tests for some quick check if everything works as it should. I have a small preference to sideline that and focus on the CI way.

---

But I think we should take a step back and discuss what is the main use-case for this because it seems to me we are trying to solve two different things.

1. Developing firmware locally and I want to run e2e tests. IMHO we do not need to have that locally as mentioned above, ideally we would have a job in FW's CI which runs Suite e2e tests. (And I think we can do that only after we stabilize the Suite tests)
2. Developing Suite and I do not want to worry about how to build some FW branch. We could solve that for example by passing Gitlab URL (or branch name maybe?) to tenv's Controller (e.g. via its Dashboard), tenv would download the appropriate artifact, and offer it.

Maybe we can discuss this on a meeting?

[gabrielKerekes]

FWIW we like to run the Connect tests with a locally built emulator while developing the Cardano app and although it's possible, it's quite complicated right now (compared to passing in the path to the emulator as it was before). Additionally it's quite error prone as there a lot of steps and for example today we weren't able to get the tests pass on a new laptop for some unknown reason (getting Transport is missing error, although the same steps worked on another)- we will take a fresh look at this tomorrow. It's also much nicer to be able to run selected tests as often as I want without having to push anything.

[grdddj]

Maybe we can discuss this on a meeting?

That would be certainly beneficial, to clearly define what we would like to have.