Setup GitHub runner for Detox tests (Apple Silicon)

[siepra]

UPDATE: Whoever takes on this ticket will either set up the existing runner on a Quiet-controlled cloud VPN and express mail it to the US, or they will migrate the tests to Github M1 runners, whichever is easier (preference for Github).

Currently tests fail sometimes because of Metro caching something, and we've had problems before of false positive passing tests due to leftover files from the previous test run. This is one reason to prefer running the tests on GH runners.

Including previous notes below

---

Here are the milestones for the full completion of mobile testing automation process:

• iOS basic tests can be run on each PR
  Ci/use self hosted runner #1931
• iOS visual regression tests can be run on each PR
• Android basic tests can be run on each PR
  Detox configuration for Android on Apple Silicon #1954
  Fix/android detox tests #1958
• Android visual regression tests can be run on each PR
• Disable Bitrise
• Run Detox tests on mobile tags
  chore: require detox before building mobile apps #1959
• Prepare long running multiplayer e2e tests

NOTE: Before setting up visual regression tests, we should try this one #1874

=======

As Bitrise turned our not to be a perfect fit for running automated e2e Detox tests, we're deciding to setup our own GitHub runner.

We need a machine that runs on Apple Silicon (arm64) and is capable of swiftly building React Native applications.
I think at least 32GB RAM and 500GB storage space is rational

[siepra]

We're waiting for the machine to be delivered to the office in Kraków

[holmesworcester]

I created #1915 after a friend recommended Amazon Device Farm for testing Android compatibility on real devices (not simulator.) The pricing seems reasonable. Is this an option?

[siepra]

Definitely worth trying

[siepra]

I'm moving to blocked as we must update Tor on mobile first

[holmesworcester]

I'm moving to blocked as we must update Tor on mobile first

Can you say a bit more about why this is blocked by updating Tor?

[holmesworcester]

As part of this work, let's document any place where the Detox tests diverge from how the app would naturally run in production.

Of the top of my head, the APK vs. APKS is one thing, and then there's something siepra mentioned about the React Native Bridge and websockets.

[siepra]

As part of this work, let's document any place where the Detox tests diverge from how the app would naturally run in production

I think Detox is self-explanatory about that https://wix.github.io/Detox/

Detox puts the actual build under test so the factors to consider are:

• Detox wrapper around the application
• Unlike human UI interaction speed
• Running on simulator

I don't know what the consequences of this are, though.

We currently put debug builds under test (debuggable, less setup complexity, probably faster), but we can do production builds if needed.

of the top of my head, the APK vs. APKS is one thing

I don't get it, can you expand?

there's something siepra mentioned about the React Native Bridge and websockets

I sure mentioned it but not in the context of Detox

[siepra]

Here's a summary of our progress so far:

1. We set up a physical, self-hosted runner for executing Detox tests for both iOS and Android
2. We failed to bring multiplayer tests to life, due to simulator incompatibility of Tor binaries

[siepra]

[GitHub hosted Apple Silicon runner]

We succeeded to use GitHub hosted Apple Silicon runner (macos-xlarge) for running iOS Detox tests.

There're problems using similar setup for Android, because of the following problem with running AVD:

HVF error: HV_UNSUPPORTED
qemu-system-aarch64: failed to initialize HVF: Invalid argument

This Stack Overflow thread suggest to re-sign qemu. Unfortunately it doesn't work on CI.

This GitHub issue implies it's a general problem and it's still pretty fresh (the latest comment is only 5 days old for the current day). We believe it's gonna be solved and it's worth to wait.

[holmesworcester]

So we will keep running tests on the local machine in the meantime? That sounds fine to me.

[holmesworcester]

@siepra I was just reading the threads you linked to, and I noticed this comment, which seems to imply that the Android emulator works on the macos-13 runner:

ReactiveCircus/android-emulator-runner#350 (comment)

Have we tried that, including with the workaround you linked to on SO? Could we use macos-13 in the meantime for Android tests?

(Moved out of blocked, but feel free to move it back if we've tried this and it's a dead end! We might as well leave this issue open though.)

[holmesworcester]

Can't do the above due to SDK target

[peterlazar1993]

I've been doing some research so maybe you'll find this useful. You cannot run a VM(android emulator) inside a VM on M1, M2 macs.

M3 macs with OSX15 (upcoming), will allow nested virtualisation - https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/