Setup E2E testing - Appium for IOS/Android #1099
Conversation
Current logic handles spaces in deviceName (ex) iPhone 13
|
@JGreenlee I also designed the script to be dynamic allowing us to test on various devices and platforms. What are your thoughts on this approach? During this initial test phase, I focused on ensuring that the appium server connects to emulators and recognizes app container. Before proceeding to the test script, I would like to discuss more about test strategy. WebdriverIO uses selector, and the selectors are differ between IOS and Android, so I am contemplating whether we should maintain separate test files for a single feature. For the sample test, I separated the selectors based on device in a single file, but I think it might become unwieldy as complexity increases. You can refer to the selectors documentation here Also, I've found some resources on running tests with GitHub Actions. I was curious about how the Appium server connects to the emulator and found the answer in this blog Emulator will be setup during the GitHub Actions execution. I will try this later! Please share any suggestions or ideas you may have regarding our e2e test 😄 |
| const getDeviceName = (platform) => { | ||
| const deviceName = process.argv.find((arg) => arg.includes('--deviceName')); | ||
| const defaultDeviceName = platform === 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33'; | ||
| return deviceName ? deviceName.split('=')[1] : defaultDeviceName; | ||
| }; | ||
|
|
||
| /** | ||
| * get Platform Version from script | ||
| * @param platform iOS or Android | ||
| */ | ||
| const getPlatformVersion = (platform) => { | ||
| const platformVersion = process.argv.find((arg) => arg.includes('--platformVersion')); | ||
| const defaultPlatformVersion = platform === 'iOS' ? '15.0' : '13'; | ||
| return platformVersion ? platformVersion.split('=')[1] : defaultPlatformVersion; | ||
| }; |
There was a problem hiding this comment.
I think the way you have it split up currently is really good. The only suggestion I might make is for these functions to be split down even further.
Perhaps getDeviceName(platform) still exists, but instead of doing the logic for 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33'; in here, we create an additional function to say getAndroidDeviceName and getiOSDeviceName, and then getDeviceName calls the appropriate one. This way we can modularize the iPhone/Android versions
Just a suggestion though! I am not too familiar with Appium or e2e testing strategies, and looking at what you've written so far looks great!!! 😊
There was a problem hiding this comment.
The flexibility provided by deviceName and platformVersion seems adequate to me.
Clever handling of this, great job!
There was a problem hiding this comment.
The one suggestion that I might have is to put these versions into a separate setup file, similar to all the other versions that we need in our build https://github.com/e-mission/e-mission-phone/blob/master/setup/export_shared_dep_versions.sh
so:
- we can upgrade without changing the code
- people can find all the versions in one place instead of wading through tests to find them.
iOSAfter opening emulator (successful) and running |
AndroidFor Android, it was successful: |
There was a problem hiding this comment.
Great, I'm so glad you have set up the architecture for this!
I think we may want to consider targeting the built app instead of the devapp (we will await input from @shankari for this). But overall, it looks like the architecture is mostly there already!
WebdriverIO uses selector, and the selectors are differ between IOS and Android, so I am contemplating whether we should maintain separate test files for a single feature. For the sample test, I separated the selectors based on device in a single file, but I think it might become unwieldy as complexity increases. You can refer to the selectors documentation here
I see the inline conditional to distinguish between android and ios selectors
const selector = driver.isAndroid
? await $('android=new UiSelector().className("android.widget.FrameLayout")')
: await $('UIATarget.localTarget().frontMostApp().mainWindow()');Maybe this distinction can just be moved to a separate function used to target the Webview.
Once inside the Webview, there are not significant differences between Android and iOS layouts and I think we should be able to target things using common IDs.
| 'appium:deviceName': getDeviceName('Android'), | ||
| 'appium:platformVersion': getPlatformVersion('Android'), | ||
| 'appium:automationName': 'UiAutomator2', | ||
| 'appium:app': join(process.cwd(), './apps/em-devapp-3.2.5.apk'), |
There was a problem hiding this comment.
I think we should discuss whether we want to run e2e tests against the devapp (in the 'serve' configuration) or against the actual e-mission app (after having built it in the 'build' configuration)
I can foresee challenges with trying to run tests through the devapp, and I also think that if we are aiming to test the true behavior of the app, the test target should be as realistic as possible (ie the fully built app)
@shankari What do you think?
There was a problem hiding this comment.
It would also negate the need to copy the devapp apk over to here.
When the app is built, the output apk appears somewhere in platforms/android/...
| const getDeviceName = (platform) => { | ||
| const deviceName = process.argv.find((arg) => arg.includes('--deviceName')); | ||
| const defaultDeviceName = platform === 'iOS' ? 'iPhone 13' : 'Pixel 3a API 33'; | ||
| return deviceName ? deviceName.split('=')[1] : defaultDeviceName; | ||
| }; | ||
|
|
||
| /** | ||
| * get Platform Version from script | ||
| * @param platform iOS or Android | ||
| */ | ||
| const getPlatformVersion = (platform) => { | ||
| const platformVersion = process.argv.find((arg) => arg.includes('--platformVersion')); | ||
| const defaultPlatformVersion = platform === 'iOS' ? '15.0' : '13'; | ||
| return platformVersion ? platformVersion.split('=')[1] : defaultPlatformVersion; | ||
| }; |
There was a problem hiding this comment.
The flexibility provided by deviceName and platformVersion seems adequate to me.
Clever handling of this, great job!
e2e-tests/config/ios.config.js
Outdated
| 'appium:deviceName': getDeviceName('iOS'), | ||
| 'appium:platformVersion': getPlatformVersion('iOS'), | ||
| 'appium:automationName': 'XCUITest', | ||
| 'appium:app': 'edu.berkeley.eecs.emission.devapp', |
There was a problem hiding this comment.
If we switched from testing the devapp to testing the built 'emission' app, this line would change to edu.berkeley.eecs.emission.
We'd need to ensure that 'emission' was installed on the simulator before running tests - hopefully this can be done through commands.
| * Gets executed before test execution begins. At this point you can access to all global | ||
| * variables like `browser`. It is the perfect place to define custom commands. | ||
| * @param {Array.<Object>} capabilities list of capabilities details | ||
| * @param {Array.<String>} specs List of spec file paths that are to be run | ||
| * @param {object} browser instance of created browser/device session | ||
| */ | ||
| // before: function (capabilities, specs) { | ||
| // }, | ||
| /** | ||
| * Runs before a WebdriverIO command gets executed. | ||
| * @param {string} commandName hook command name | ||
| * @param {Array} args arguments that command would receive | ||
| */ | ||
| // beforeCommand: function (commandName, args) { | ||
| // }, | ||
| /** | ||
| * Hook that gets executed before the suite starts | ||
| * @param {object} suite suite details | ||
| */ | ||
| // beforeSuite: function (suite) { | ||
| // }, | ||
| /** | ||
| * Function to be executed before a test (in Mocha/Jasmine) starts. | ||
| */ | ||
| // beforeTest: function (test, context) { | ||
| // }, | ||
| /** | ||
| * Hook that gets executed _before_ a hook within the suite starts (e.g. runs before calling | ||
| * beforeEach in Mocha) | ||
| */ | ||
| // beforeHook: function (test, context, hookName) { | ||
| // }, | ||
| /** | ||
| * Hook that gets executed _after_ a hook within the suite starts (e.g. runs after calling | ||
| * afterEach in Mocha) | ||
| */ | ||
| // afterHook: function (test, context, { error, result, duration, passed, retries }, hookName) { | ||
| // }, | ||
| /** | ||
| * Function to be executed after a test (in Mocha/Jasmine only) | ||
| * @param {object} test test object | ||
| * @param {object} context scope object the test was executed with | ||
| * @param {Error} result.error error object in case the test fails, otherwise `undefined` | ||
| * @param {*} result.result return object of test function | ||
| * @param {number} result.duration duration of test | ||
| * @param {boolean} result.passed true if test has passed, otherwise false | ||
| * @param {object} result.retries information about spec related retries, e.g. `{ attempts: 0, limit: 0 }` | ||
| */ | ||
| // afterTest: function(test, context, { error, result, duration, passed, retries }) { | ||
| // }, | ||
|
|
||
| /** | ||
| * Hook that gets executed after the suite has ended | ||
| * @param {object} suite suite details | ||
| */ | ||
| // afterSuite: function (suite) { | ||
| // }, | ||
| /** | ||
| * Runs after a WebdriverIO command gets executed | ||
| * @param {string} commandName hook command name | ||
| * @param {Array} args arguments that command would receive | ||
| * @param {number} result 0 - command success, 1 - command error | ||
| * @param {object} error error object if any | ||
| */ | ||
| // afterCommand: function (commandName, args, result, error) { | ||
| // }, | ||
| /** | ||
| * Gets executed after all tests are done. You still have access to all global variables from | ||
| * the test. | ||
| * @param {number} result 0 - test pass, 1 - test fail | ||
| * @param {Array.<Object>} capabilities list of capabilities details | ||
| * @param {Array.<String>} specs List of spec file paths that ran | ||
| */ | ||
| // after: function (result, capabilities, specs) { | ||
| // }, | ||
| /** | ||
| * Gets executed right after terminating the webdriver session. | ||
| * @param {object} config wdio configuration object | ||
| * @param {Array.<Object>} capabilities list of capabilities details | ||
| * @param {Array.<String>} specs List of spec file paths that ran | ||
| */ | ||
| // afterSession: function (config, capabilities, specs) { | ||
| // }, | ||
| /** | ||
| * Gets executed after all workers got shut down and the process is about to exit. An error | ||
| * thrown in the onComplete hook will result in the test run failing. | ||
| * @param {object} exitCode 0 - success, 1 - fail | ||
| * @param {object} config wdio configuration object | ||
| * @param {Array.<Object>} capabilities list of capabilities details | ||
| * @param {<Object>} results object containing test results | ||
| */ | ||
| // onComplete: function(exitCode, config, capabilities, results) { | ||
| // }, | ||
| /** | ||
| * Gets executed when a refresh happens. | ||
| * @param {string} oldSessionId session ID of the old session | ||
| * @param {string} newSessionId session ID of the new session | ||
| */ | ||
| // onReload: function(oldSessionId, newSessionId) { | ||
| // } |
There was a problem hiding this comment.
I see that we aren't using hooks now, but it's good to keep this around since we may need them later.
| describe('Connect test', () => { | ||
| it('should call app successfully', async () => { | ||
| const selector = driver.isAndroid | ||
| ? await $('android=new UiSelector().className("android.widget.FrameLayout")') |
There was a problem hiding this comment.
Can you explain to me what $ does in this instance? Is this how we select elements? (sort of like jQuery?)
There was a problem hiding this comment.
Yes, $ command is a short and handy way in order to fetch a single element on the page.
You can check more information here
| "appium-ios": "npx wdio ./e2e-tests/config/ios.config.js --", | ||
| "appium-android": "npx wdio ./e2e-tests/config/android.config.js --" |
There was a problem hiding this comment.
If we switch to doing e2e tests against a built app instead of the devapp, these will instead be in package.cordovabuild.json.
| "@wdio/appium-service": "^8.22.1", | ||
| "@wdio/cli": "^8.22.1", | ||
| "@wdio/local-runner": "^8.22.1", | ||
| "@wdio/mocha-framework": "^8.22.0", | ||
| "@wdio/spec-reporter": "^8.21.0", | ||
| "appium": "^2.2.1", | ||
| "appium-xcuitest-driver": "^5.8.2", | ||
| "appium-uiautomator2-driver": "^2.34.1" |
There was a problem hiding this comment.
^ (along with the dependencies)
There was a problem hiding this comment.
Maybe we could keep tests as e2e-tests/sample.spec.js instead of nesting them in a specs folder?
I've seen the testname.spec.js convention used a lot in other projects
|
@JGreenlee I am currently investigating the differences between the instances when the installation of Additionally, as you pointed out, the |
|
@JGreenlee @jiji14 thanks so much for doing this! On the other hand, before we release a new version, I would like to run the tests against the built version. For GitHub actions, we can run the devapp version as part of the UI build ( |
|
I thought that we would only be running the e2e tests in GitHub actions, not manually. I'm anticipating that once we have a comprehensive suite of e2e tests, they could take a very long time to run (potentially 1+ hours?) and I don't think it will be practical to run them all locally.
The (Appium) e2e tests are separate from the (Jest) unit/component tests; I don't think there is any potential for overlap or communication between them. |
I understand that the appium test framework is different from Jest, but in terms of the functionality that we want to test, I think that there might indeed be overlap. For example, we would probably want to have an e2e test which loads data into the label screen, but runs through the full stack of UI -> native -> server to retrieve the data instead of looking only at mock results.
From experience on the server side, this is not a great idea. This will work for validation, but once the tests break, it is hard to debug tests that are run using GitHub actions, or to verify that the fixes have worked. It would be best if we could also run individual e2e tests locally. |
|
I resolved the IOS test issue by running the
Here is what happened to me. I got the While there are ways to manually install I think we can simply run the IOS test by manually installing |
|
For the next step, Jack will handle the iOS setup, while I focus on developing the Android Appium test workflow in GitHub Actions. |
Setup E2E testing - Appium for IOS/Android
Related Issue
Considerations for Integration and End-to-End Testing Strategies
Testing Overview
I've configured
WebdriverIOandAppiumfor E2E testing.WebdriverIO, an open-source testing utility for Node.js, seamlessly integrates withAppium, allowing us to write test scripts for both IOS and Android platforms. We have separate configuration files for iOS and Android, and we need to run tests for each platform separately. Test scripts will be written inWebdriverIOusingMocha. (Unfortunately,Jestwas not in options, andMochawas the best alternative due to its similarity.)Check out this article for a comprehensive guide on
AppiumandWebdriverIOintegration.How to run test locally
Because we need to test different devices and platform versions, I made the script dynamic. When you run the test, you can pass the device name and platform version as follows:
If you encounter this issue,
Try this,
Resource: Stack Overflow
Make sure
apkfile is in your local repository (e-mission-phone > apps > em-devapp-3.2.5.apk)Test result
To do
Currently, the test checks if the Appium server can run successfully, connects to the emulator, and verifies the visibility of the app container.
Explore setting up E2E tests on GitHub Actions. Investigate how Appium can access the emulator or real devices in the GitHub Actions environment.