-
Notifications
You must be signed in to change notification settings - Fork 38
feat: Maestro E2E test suite POC with GitHub Actions iOS + Android CI #1911
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
f955765
cca8940
c095adf
09b4545
0e96df1
66d1ae0
a72c8b6
ba680e5
1f3a067
adc65af
84e63eb
91ba444
517f532
c84afd6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,284 @@ | ||
| name: E2E Tests (Maestro) | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main, development] | ||
| paths: | ||
| - "apps/expo/**" | ||
| - ".maestro/**" | ||
| - ".github/workflows/e2e-tests.yml" | ||
| # Note: Using `pull_request` (not `pull_request_target`) so forked PRs get | ||
| # CI feedback on their own code. Secrets are unavailable for forks, so | ||
| # the job is skipped via the `if` condition on the job below. | ||
| pull_request: | ||
| branches: [main, development] | ||
| paths: | ||
| - "apps/expo/**" | ||
| - ".maestro/**" | ||
| - ".github/workflows/e2e-tests.yml" | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| env: | ||
| MAESTRO_VERSION: 1.40.0 | ||
|
|
||
| jobs: | ||
| ios-e2e: | ||
| name: iOS E2E Tests | ||
| runs-on: macos-15 | ||
| timeout-minutes: 120 | ||
| # Skip on forked PRs — secrets are not available in forks | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
|
|
||
| env: | ||
| TEST_EMAIL: ${{ secrets.E2E_TEST_EMAIL }} | ||
| TEST_PASSWORD: ${{ secrets.E2E_TEST_PASSWORD }} | ||
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Bun | ||
| uses: oven-sh/setup-bun@v1 | ||
| with: | ||
| bun-version: latest | ||
|
|
||
| - name: Install dependencies | ||
| env: | ||
| PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN: ${{ secrets.PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN }} | ||
| run: bun install | ||
|
|
||
| - name: Setup Expo | ||
| uses: expo/expo-github-action@v8 | ||
| with: | ||
| eas-version: latest | ||
| token: ${{ secrets.EXPO_TOKEN }} | ||
|
|
||
| - name: Setup Java | ||
| uses: actions/setup-java@v4 | ||
| with: | ||
| distribution: temurin | ||
| java-version: "17" | ||
|
|
||
| - name: Install Maestro CLI | ||
| run: | | ||
| export MAESTRO_VERSION="${{ env.MAESTRO_VERSION }}" | ||
| curl -Ls "https://get.maestro.mobile.dev" | bash | ||
| echo "${HOME}/.maestro/bin" >> "${GITHUB_PATH}" | ||
|
|
||
| - name: Build iOS app for simulator | ||
| run: | | ||
| cd apps/expo | ||
| mkdir -p build | ||
| eas build \ | ||
| --platform ios \ | ||
| --profile e2e \ | ||
| --non-interactive \ | ||
| --local \ | ||
| --output ./build/PackRat-sim.tar.gz | ||
| env: | ||
|
Comment on lines
+74
to
+84
|
||
| EXPO_TOKEN: ${{ secrets.EXPO_TOKEN }} | ||
| PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN: ${{ secrets.PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN }} | ||
|
|
||
| - name: Extract iOS simulator build | ||
| run: | | ||
| mkdir -p apps/expo/build/extracted | ||
| tar -xzf apps/expo/build/PackRat-sim.tar.gz -C apps/expo/build/extracted | ||
| APP_PATH=$(find apps/expo/build/extracted -name "*.app" -maxdepth 3 | head -1) | ||
| if [ -z "$APP_PATH" ]; then | ||
| echo "::error::No .app bundle found in simulator build archive" | ||
| exit 1 | ||
| fi | ||
| echo "APP_PATH=$APP_PATH" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Boot iOS Simulator | ||
| run: | | ||
| # Find the latest available iOS runtime (prefer 18, fall back to 17, then any available iOS runtime) | ||
| IOS_RUNTIME=$(xcrun simctl list runtimes --json \ | ||
| | python3 -c " | ||
| import sys, json | ||
| data = json.load(sys.stdin) | ||
| runtimes = data.get('runtimes', []) | ||
| available = [r for r in runtimes if r.get('isAvailable', False) and r.get('platform', r.get('name','')).startswith('iOS')] | ||
| # Fallback: also match by name containing 'iOS' in case 'platform' key is absent | ||
| if not available: | ||
| available = [r for r in runtimes if r.get('isAvailable', False) and 'iOS' in r.get('name','')] | ||
| ios18 = [r for r in available if 'iOS 18' in r.get('name','')] | ||
| ios17 = [r for r in available if 'iOS 17' in r.get('name','')] | ||
| chosen = ios18 or ios17 or available | ||
| if not chosen: | ||
| print('', file=sys.stderr) | ||
| sys.exit(1) | ||
| print(chosen[-1]['identifier']) | ||
| ") | ||
| if [ -z "$IOS_RUNTIME" ]; then | ||
| echo "::error::No available iOS runtime found on this runner" | ||
| exit 1 | ||
| fi | ||
| # Find iPhone 15 device type (base or Pro) | ||
| DEVICE_TYPE=$(xcrun simctl list devicetypes --json \ | ||
| | python3 -c " | ||
| import sys, json | ||
| types = json.load(sys.stdin)['devicetypes'] | ||
| iphone15 = [d for d in types if 'iPhone 15' in d.get('name','') and 'Pro' not in d.get('name','') and 'Plus' not in d.get('name','') and 'Max' not in d.get('name','')] | ||
| fallback = [d for d in types if 'iPhone 15' in d.get('name','')] | ||
| chosen = (iphone15 or fallback) | ||
| print(chosen[0]['identifier'] if chosen else 'com.apple.CoreSimulator.SimDeviceType.iPhone-15') | ||
| ") | ||
| echo "Using runtime: $IOS_RUNTIME" | ||
| echo "Using device type: $DEVICE_TYPE" | ||
| DEVICE_ID=$(xcrun simctl create "PackRat-E2E" "$DEVICE_TYPE" "$IOS_RUNTIME") | ||
| xcrun simctl boot "$DEVICE_ID" | ||
| # Wait for the simulator to be fully booted before continuing | ||
| xcrun simctl bootstatus "$DEVICE_ID" -b | ||
| echo "SIMULATOR_UDID=$DEVICE_ID" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Install app on simulator | ||
| run: | | ||
| xcrun simctl install "$SIMULATOR_UDID" "$APP_PATH" | ||
|
|
||
|
Comment on lines
+135
to
+144
|
||
| - name: Run Maestro E2E tests | ||
| run: | | ||
| mkdir -p test-results | ||
| maestro --device "$SIMULATOR_UDID" test \ | ||
| --format junit \ | ||
| --output test-results/maestro-results.xml \ | ||
| .maestro/config.yaml | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| env: | ||
| TEST_EMAIL: ${{ secrets.E2E_TEST_EMAIL }} | ||
| TEST_PASSWORD: ${{ secrets.E2E_TEST_PASSWORD }} | ||
| TRIP_NAME: E2E-Trip-${{ github.run_id }} | ||
| PACK_NAME: E2E-Pack-${{ github.run_id }} | ||
|
|
||
| - name: Upload test results | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: ios-e2e-results | ||
| path: test-results/ | ||
| retention-days: 30 | ||
|
|
||
| - name: Upload Maestro screenshots and videos on failure | ||
| if: failure() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: ios-e2e-failure-artifacts | ||
| path: ~/.maestro/tests/ | ||
| retention-days: 7 | ||
|
|
||
| - name: Shutdown simulator | ||
| if: always() | ||
| run: | | ||
| xcrun simctl shutdown "$SIMULATOR_UDID" || true | ||
| xcrun simctl delete "$SIMULATOR_UDID" || true | ||
|
|
||
| android-e2e: | ||
| name: Android E2E Tests | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 120 | ||
| # Skip on forked PRs — secrets are not available in forks | ||
| if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository | ||
|
|
||
| env: | ||
| TEST_EMAIL: ${{ secrets.E2E_TEST_EMAIL }} | ||
| TEST_PASSWORD: ${{ secrets.E2E_TEST_PASSWORD }} | ||
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Bun | ||
| uses: oven-sh/setup-bun@v1 | ||
| with: | ||
| bun-version: latest | ||
|
|
||
| - name: Install dependencies | ||
| env: | ||
| PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN: ${{ secrets.PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN }} | ||
| run: bun install | ||
|
|
||
| - name: Setup Expo | ||
| uses: expo/expo-github-action@v8 | ||
| with: | ||
| eas-version: latest | ||
| token: ${{ secrets.EXPO_TOKEN }} | ||
|
|
||
| - name: Setup Java | ||
| uses: actions/setup-java@v4 | ||
| with: | ||
| distribution: temurin | ||
| java-version: "17" | ||
|
|
||
| - name: Install Maestro CLI | ||
| run: | | ||
| export MAESTRO_VERSION="${{ env.MAESTRO_VERSION }}" | ||
| curl -Ls "https://get.maestro.mobile.dev" | bash | ||
| echo "${HOME}/.maestro/bin" >> "${GITHUB_PATH}" | ||
|
|
||
| - name: Build Android APK for emulator | ||
| run: | | ||
| cd apps/expo | ||
| mkdir -p build | ||
| eas build \ | ||
| --platform android \ | ||
| --profile e2e \ | ||
| --non-interactive \ | ||
| --local \ | ||
| --output ./build/PackRat.apk | ||
| env: | ||
| EXPO_TOKEN: ${{ secrets.EXPO_TOKEN }} | ||
| PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN: ${{ secrets.PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN }} | ||
|
|
||
| - name: Enable KVM for hardware acceleration | ||
| run: | | ||
| echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ | ||
| | sudo tee /etc/udev/rules.d/99-kvm4all.rules | ||
| sudo udevadm control --reload-rules | ||
| sudo udevadm trigger --name-match=kvm | ||
|
|
||
| - name: Run Maestro E2E tests on Android emulator | ||
| uses: reactivecircus/android-emulator-runner@v2.34.0 | ||
| with: | ||
| api-level: 34 | ||
| target: google_apis | ||
| arch: x86_64 | ||
| profile: pixel_6 | ||
| avd-name: PackRat-E2E | ||
| force-avd-creation: false | ||
| emulator-options: >- | ||
| -no-snapshot-save -no-window -gpu swiftshader_indirect | ||
| -noaudio -no-boot-anim -camera-back none | ||
| disable-animations: true | ||
| script: | | ||
| mkdir -p test-results | ||
| adb install apps/expo/build/PackRat.apk | ||
| maestro test \ | ||
| --format junit \ | ||
| --output test-results/maestro-results.xml \ | ||
| .maestro/config-android.yaml | ||
| env: | ||
| TEST_EMAIL: ${{ secrets.E2E_TEST_EMAIL }} | ||
| TEST_PASSWORD: ${{ secrets.E2E_TEST_PASSWORD }} | ||
| TRIP_NAME: E2E-Trip-${{ github.run_id }} | ||
| PACK_NAME: E2E-Pack-${{ github.run_id }} | ||
|
|
||
| - name: Upload test results | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: android-e2e-results | ||
| path: test-results/ | ||
| retention-days: 30 | ||
|
|
||
| - name: Upload Maestro screenshots and videos on failure | ||
| if: failure() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: android-e2e-failure-artifacts | ||
| path: ~/.maestro/tests/ | ||
| retention-days: 7 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # PackRat Maestro E2E Tests | ||
|
|
||
| This directory contains end-to-end tests for the PackRat mobile app using [Maestro](https://maestro.mobile.dev). | ||
|
|
||
| ## Structure | ||
|
|
||
| ```text | ||
| .maestro/ | ||
| ├── config.yaml # Maestro suite configuration & flow order | ||
| └── flows/ | ||
| ├── setup/ | ||
| │ └── clear-state.yaml # Clears app state before test suite | ||
| ├── auth/ | ||
| │ ├── login-flow.yaml # Sign in with email and password | ||
| │ └── logout-flow.yaml # Sign out from the app | ||
| ├── trips/ | ||
| │ └── create-trip-flow.yaml # Create a new trip | ||
| └── packs/ | ||
| └── create-pack-flow.yaml # Create a new pack | ||
| ``` | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| 1. **Install Maestro CLI**: | ||
| ```bash | ||
| curl -Ls "https://get.maestro.mobile.dev" | bash | ||
| ``` | ||
|
|
||
| 2. **Set up environment variables** for test credentials: | ||
| ```bash | ||
| export TEST_EMAIL="your-test-account@example.com" | ||
| export TEST_PASSWORD="your-test-password" | ||
| ``` | ||
|
|
||
| 3. **Build and install the app** on a simulator/device (run from `apps/expo/`): | ||
| ```bash | ||
| eas build --platform ios --profile e2e --local --output ./build/PackRat-sim.tar.gz | ||
| mkdir -p ./build/extracted | ||
| tar -xzf ./build/PackRat-sim.tar.gz -C ./build/extracted | ||
| xcrun simctl install booted ./build/extracted/*.app | ||
| ``` | ||
|
|
||
| ## Running Tests | ||
|
|
||
| ### Run all flows | ||
| ```bash | ||
| maestro test .maestro/config.yaml | ||
| ``` | ||
|
|
||
| ### Run a single flow | ||
| ```bash | ||
| maestro test .maestro/flows/auth/login-flow.yaml | ||
| ``` | ||
|
|
||
| ### Run with JUnit output (for CI) | ||
| ```bash | ||
| maestro test --format junit --output test-results.xml .maestro/config.yaml | ||
| ``` | ||
|
|
||
| ### Run on a specific simulator | ||
| ```bash | ||
| maestro --device <UDID> test .maestro/config.yaml | ||
| ``` | ||
|
|
||
| ## CI/CD | ||
|
|
||
| Tests run automatically via GitHub Actions (`.github/workflows/e2e-tests.yml`) on: | ||
| - Every push to `main` or `development` that touches `apps/expo/**`, `.maestro/**`, or `.github/workflows/e2e-tests.yml` | ||
| - Every pull request targeting `main` or `development` that touches the same paths | ||
|
|
||
| > **Note:** Forked pull requests are skipped because repository secrets are not available. | ||
|
|
||
| ### Required Secrets | ||
|
|
||
| Configure these GitHub repository secrets for CI: | ||
|
|
||
| | Secret | Description | | ||
| |--------|-------------| | ||
| | `EXPO_TOKEN` | Expo access token for EAS builds | | ||
| | `E2E_TEST_EMAIL` | Email address of the E2E test account | | ||
| | `E2E_TEST_PASSWORD` | Password of the E2E test account | | ||
| | `PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN` | GitHub token for private package access | | ||
|
|
||
| ## Writing New Tests | ||
|
|
||
| Flows are standard YAML files following the [Maestro flow syntax](https://maestro.mobile.dev/api-reference/commands). | ||
|
|
||
| Key conventions: | ||
| - All flows start with `appId: com.andrewbierman.packrat` | ||
| - Prefer `testID` / `accessibilityIdentifier` selectors first (e.g., `tapOn: { id: "submitButton" }`); use `text` to match an element's iOS `accessibilityLabel` or visible text (e.g., `tapOn: { text: "Submit" }`); `accessibilityLabel` is **not** a valid Maestro selector key | ||
| - Use `waitForAnimationToEnd` after navigation actions | ||
| - Use `runFlow: { when: { visible: ... } }` for conditional steps | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| - Use environment variables (e.g., `${TRIP_NAME}`) for entity names to keep each test run unique | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - **Element not found**: Run `maestro studio` for interactive element inspection | ||
| - **Flaky tests**: Add `waitForAnimationToEnd` before assertions | ||
| - **Simulator issues**: Ensure the simulator is booted before running tests | ||
Uh oh!
There was an error while loading. Please reload this page.