feat: Maestro E2E test suite POC with GitHub Actions iOS + Android CI#1911
Conversation
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
|
It looks like your comment came through empty — did you have a specific question or request about the E2E test implementation? Happy to help with any follow-up. |
|
This comment also came through empty. If you have a specific question or change request about the E2E test implementation, please share the details and I'll take care of it. |
|
@copilot build failing |
|
@coderabbitai Please review this PR for code quality, best practices, and potential issues. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 11826
🔴 Critical — Will Break the CI Run1. EAS local iOS build does not produce a
|
| Severity | Issue |
|---|---|
| 🔴 Critical | EAS local build outputs .tar.gz/.ipa, not .app — install step will fail |
| 🔴 Critical | id:-based taps in trip/pack flows won't match (no testIDs in codebase) |
| 🔴 Critical | test-results/ directory not created before Maestro output |
| 🟡 High | Java setup step is unnecessary for iOS builds |
| 🟡 High | No guard for empty Python runtime/device selection output |
| 🟠 Medium | bun-version: latest and eas-version: latest should be pinned |
| 🟠 Medium | No dependency caching |
| 🟠 Medium | 60-min timeout too tight for EAS local builds |
| 🔵 Low | Redundant clearState in clear-state.yaml |
| 🔵 Low | Logout dialog confirmation race condition |
| 🔵 Low | Missing minVersion in config.yaml |
The three critical items will need to be resolved for the CI pipeline to succeed. Happy to help generate the specific diffs for any of them.
[full_review]
🧠 Learnings used
Learnt from: CR
Repo: PackRat-AI/PackRat PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2026-03-01T10:08:42.658Z
Learning: Configure required CI/CD environment variables: PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN for GitHub package access, Cloudflare API tokens for API deployment, and Expo credentials for mobile builds
There was a problem hiding this comment.
Pull request overview
Adds a proof-of-concept Maestro E2E suite for the Expo mobile app and wires it into GitHub Actions to run on an iOS simulator, targeting core flows (login → create trip → create pack → logout).
Changes:
- Introduces a Maestro test suite config plus setup/auth/trips/packs flow YAMLs under
.maestro/. - Adds documentation for running Maestro locally and required CI secrets.
- Adds an iOS CI workflow that builds the app, boots a simulator, installs the app, runs Maestro, and uploads artifacts.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
.maestro/config.yaml |
Defines suite flow discovery and explicit execution order. |
.maestro/flows/setup/clear-state.yaml |
Attempts to reset app state for deterministic runs. |
.maestro/flows/auth/login-flow.yaml |
Automates email/password login and asserts tab visibility. |
.maestro/flows/auth/logout-flow.yaml |
Automates logout from the Profile tab and asserts auth screen. |
.maestro/flows/trips/create-trip-flow.yaml |
Navigates to Trips and creates a trip. |
.maestro/flows/packs/create-pack-flow.yaml |
Navigates to Packs and creates a pack. |
.maestro/README.md |
Documents structure, prerequisites, and CI usage. |
.github/workflows/e2e-tests.yml |
Adds macOS iOS-simulator CI job to build/install/run Maestro and upload results/artifacts. |
You can also share your feedback on Copilot code review. Take the survey.
| - clearState | ||
| - launchApp: | ||
| clearState: true |
There was a problem hiding this comment.
clear-state.yaml runs both clearState and launchApp with clearState: true, which duplicates the same intent and increases suite time. Consider keeping just the launchApp call with clearState: true (or just clearState) to avoid redundant work.
| - scrollUntilVisible: | ||
| element: | ||
| text: "Sign Out" | ||
| direction: DOWN | ||
|
|
||
| # Tap Sign Out | ||
| - tapOn: | ||
| text: "Sign Out" | ||
|
|
There was a problem hiding this comment.
logout-flow.yaml is looking for and tapping a "Sign Out" button, but the Profile tab’s logout button text in the app is "Log Out" (t('auth.logOut')). As written, scrollUntilVisible/tapOn will never find the element on iOS, so the flow will fail. Update the flow to target the actual "Log Out" label and handle the post-logout alert (e.g., choose "Stay logged out" / "Sign-in again") before asserting the auth screen.
| - runFlow: | ||
| when: | ||
| notVisible: | ||
| text: "New Trip" | ||
| commands: | ||
| - tapOn: | ||
| id: "create-trip-button" | ||
|
|
There was a problem hiding this comment.
This flow falls back to tapOn: { id: "create-trip-button" }, but there is no matching accessibility identifier/testID in the app for the Trips header "+" button (it’s a plain <Pressable> wrapping an icon). On non-empty accounts (server already has trips), the "Create New Trip" empty-state button won’t be visible and this fallback will fail. Either add a stable accessibilityLabel/testID for the header create button in the app code and reference that here, or change the flow strategy to not depend on header UI when the list isn’t empty.
| - runFlow: | ||
| when: | ||
| notVisible: | ||
| text: "Create Pack" | ||
| commands: | ||
| - tapOn: | ||
| id: "create-pack-button" | ||
|
|
There was a problem hiding this comment.
This flow falls back to tapOn: { id: "create-pack-button" }, but there is no matching accessibility identifier/testID in the app for the Packs header "+" button (it’s a plain <Pressable> wrapping an icon). If the test account already has packs (server-side), the empty-state "Create New Pack" button won’t appear and this fallback will fail. Either add a stable accessibilityLabel/testID in the app code for the header create button and reference it here, or adjust the flow to a selector that exists regardless of list emptiness.
| # Submit the form | ||
| - tapOn: | ||
| text: "Create Pack" | ||
|
|
||
| - waitForAnimationToEnd |
There was a problem hiding this comment.
On the pack creation screen, the navigation title is also "Create Pack" (from the Expo Router Stack options), so tapOn: { text: "Create Pack" } can match the header title instead of the submit button, making the test flaky. Prefer a more specific selector (e.g., target the submit button via an accessibility id/label) so the tap reliably hits the form submit control.
| - name: Build iOS app for simulator | ||
| run: | | ||
| cd apps/expo | ||
| eas build \ | ||
| --platform ios \ | ||
| --profile preview \ | ||
| --non-interactive \ | ||
| --local \ | ||
| --output ./build/PackRat.app | ||
| env: |
There was a problem hiding this comment.
The workflow builds with --profile preview but apps/expo/eas.json does not configure an iOS simulator build (no ios.simulator: true in that profile). If EAS outputs an .ipa (device build), xcrun simctl install will fail because it expects a .app. Consider adding a dedicated EAS build profile for simulator builds and using it here (or adjust the workflow to handle the produced artifact type).
| DEVICE_ID=$(xcrun simctl create "PackRat-E2E" "$DEVICE_TYPE" "$IOS_RUNTIME") | ||
| xcrun simctl boot "$DEVICE_ID" | ||
| echo "SIMULATOR_UDID=$DEVICE_ID" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Install app on simulator | ||
| run: | | ||
| xcrun simctl install "$SIMULATOR_UDID" apps/expo/build/PackRat.app | ||
|
|
There was a problem hiding this comment.
After xcrun simctl boot, the workflow immediately installs the app. simctl boot is not guaranteed to wait until the simulator is fully booted, which can cause intermittent install / Maestro startup failures. Add an explicit wait step (e.g., xcrun simctl bootstatus "$DEVICE_ID" -b) before installing/running tests.
| # Find the latest available iOS 17+ runtime | ||
| IOS_RUNTIME=$(xcrun simctl list runtimes --json \ | ||
| | python3 -c " | ||
| import sys, json | ||
| runtimes = json.load(sys.stdin)['runtimes'] | ||
| ios17 = [r for r in runtimes if r.get('isAvailable') and 'iOS 17' in r.get('name','')] | ||
| ios18 = [r for r in runtimes if r.get('isAvailable') and 'iOS 18' in r.get('name','')] | ||
| chosen = (ios18 or ios17) | ||
| print(chosen[-1]['identifier'] if chosen else '') | ||
| ") |
There was a problem hiding this comment.
IOS_RUNTIME can end up empty if the runner image doesn’t include an iOS 17/18 runtime matching the name filter. In that case simctl create will fail with a hard-to-debug error. Add a guard that errors with a clear message when IOS_RUNTIME is empty, or broaden the selection logic to fall back to the newest available iOS runtime regardless of major version.
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Fixed in c095adf. Root cause: The Fix:
|
|
@coderabbitai Copilot has pushed updates addressing your previous feedback. Please re-review this PR for any remaining issues or approve if it looks good. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chainLet me fetch the current state of all relevant files. 🏁 Script executed: Length of output: 9724
✅ Resolved
🔴 Still Critical1.
|
| Severity | Issue | Status |
|---|---|---|
| 🔴 Critical | test-results/ dir not created |
Unresolved |
| 🔴 Critical | id: fallback taps still present |
Unresolved |
| 🟡 High | No bash guard on empty runtime/device vars | Unresolved |
| 🟡 High | Unnecessary Java 17 setup | Unresolved |
| 🟠 Medium | APP_PATH stored as relative path |
New |
| 🟠 Medium | bun-version/eas-version unpinned |
Unresolved |
| 🟠 Medium | No dependency caching | Unresolved |
| 🟠 Medium | 60-min timeout too tight | Unresolved |
| ✅ Resolved | EAS .tar.gz extraction with APP_PATH validation |
Fixed |
| ✅ Resolved | eas.json e2e simulator profile |
Fixed |
| ✅ Resolved | Text-based taps as primary path in trip/pack flows | Fixed |
The test-results/ directory creation and the runtime selection empty-string guards are the two most likely to cause an immediate CI failure — I'd prioritize those along with removing the Java step. Happy to generate the full diffs for any of these.
🧠 Learnings used
Learnt from: CR
Repo: PackRat-AI/PackRat PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2026-03-01T10:08:42.658Z
Learning: Configure required CI/CD environment variables: PACKRAT_NATIVEWIND_UI_GITHUB_TOKEN for GitHub package access, Cloudflare API tokens for API deployment, and Expo credentials for mobile builds
|
@copilot CodeRabbit has reviewed this PR and provided feedback. Please address the review comments and push updates. Once you've made the changes, the monitor will ask CodeRabbit to re-review. |
|
@copilot let's add android maestro tests also pls |
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Done in
|
|
@copilot lets use an enum for test ids |
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Done in export enum TestIds {
SignInEmailButton = 'sign-in-email-button',
EmailInput = 'email-input',
PasswordInput = 'password-input',
ContinueButton = 'continue-button',
CreateTripButton = 'create-trip-button',
SubmitTripButton = 'submit-trip-button',
CreatePackButton = 'create-pack-button',
SubmitPackButton = 'submit-pack-button',
}The string values are unchanged, so the Maestro YAML flows continue to work without modification. |
e2esimulator build profile toeas.jsone2e-tests.ymlto use--profile e2e, extract.tar.gz, validate.apppathPACKRAT_NATIVEWIND_UI_GITHUB_TOKENto EAS local build stepmacos-14→macos-15(Xcode 16)testIDselectors to auth screens; updatelogin-flow.yamlto useid:selectorsappId: com.andrewbierman.packrattoconfig.yaml(was causingConfig Field Required: appIderror)accessibilityLabelMaestro selector key with correct guidance (id→accessibilityIdentifier,text→accessibilityLabel)login-flow.yaml: Replace text-based final assertions with stableid: "create-pack-button"testID selectorandroid-e2eonubuntu-latestwith KVM +reactivecircus/android-emulator-runner).maestro/config-android.yamlwithappId: com.packratai.mobilefor Android suiteappIdheaders so individual flows inheritappIdfrom whichever suite config drives them (iOS or Android)android: { buildType: "apk" }to thee2eEAS build profileMAESTRO_VERSIONto workflow-level env var shared by both iOS and Android jobsTestIdsenum inapps/expo/lib/testIds.ts; update all components to reference enum values instead of inlinetestIDstringsOriginal prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
Summary by CodeRabbit
Tests
Documentation
Chores