|
| 1 | +--- |
| 2 | +title: Mobile App Testing Frameworks Overview |
| 3 | +description: "Mobile App Testing Frameworks Overview" |
| 4 | +keywords: testcloud |
| 5 | +author: glennwester |
| 6 | +ms.author: glwest |
| 7 | +ms.date: 4/6/2018 |
| 8 | +ms.topic: article |
| 9 | +ms.assetid: 9cdf95db-1ad5-4541-a21c-c20492d4607f |
| 10 | +ms.service: vs-appcenter |
| 11 | +ms.custom: test |
| 12 | +--- |
| 13 | +## Mobile App Testing Frameworks Overview |
| 14 | + |
| 15 | +## Introduction |
| 16 | + |
| 17 | +After delivering support for next major release of the iOS and Android operating systems, Microsoft will discontinue our contributions to developing Calabash, the open-source mobile app testing tool. We hope that the community will continue to fully adopt and maintain it. |
| 18 | + |
| 19 | +As part of our transition on the development of Calabash, we've provided an overview of mobile app UI and end-to-end testing frameworks as a starting point for teams who are looking to re-evaluate their testing strategy. This document is written to aid such an evaluation. |
| 20 | + |
| 21 | +## The Future of Calabash |
| 22 | +Microsoft is committed to ensuring that Calabash works up until and including the next minor releases of the iOS and Android OS versions, that is, iOS 11.x and Android O / Android 8.x. After securing those releases, we will discontinue dedicated engineering contributions. Given the open source nature of Calabash, we hope the community will step up and continue to maintain and develop the tool. If you're interested in staying with Calabash, you'll want to take note to see if an individual or group steps up to take over maintenance. You may also see this as an opportunity to take matters into your own hands and devote engineering resources to maintain Calabash. If you're interested in taking over Calabash stewardship, please contact <[email protected]>. |
| 23 | + |
| 24 | +Calabash will continue to be supported in Visual Studio App Center and Xamarin Test Cloud up until and including iOS 11.x and Android 8.x. At this time, the tool will become a 3rd party testing framework supported in Test Cloud. |
| 25 | + |
| 26 | +Still have questions about how it works? Read on for our FAQ: |
| 27 | + |
| 28 | +**Q: Will we be able to continue to use Calabash to run tests on my devices and simulators? Will Calabash continue to be supported in Xamarin Test Cloud & Visual Studio App Center?** |
| 29 | + |
| 30 | +**A:** Yes, you can continue to use Calabash. There is a risk that the community will not find a maintainer and that Calabash might not support OS versions greater than iOS 11.x and Android 8.x. |
| 31 | + |
| 32 | +**Q: Will Microsoft release the entire Calabash stack to the OSS community?** |
| 33 | + |
| 34 | +**A:** Microsoft will release the source code to DeviceAgent.iOS and iOSDeviceManager as open source, so that Calabash can be considered fully open source. |
| 35 | + |
| 36 | +**Q: Xamarin.UITest is based on Calabash – what does Calabash deprecation mean for Xamarin.UITest?** |
| 37 | + |
| 38 | +**A:** While it is correct that Xamarin.UITest uses some components from the Calabash tool-chain, it's important to stress that Xamarin.UITest is still fully maintained and actively being developed. Xamarin.UITest aspires to be the de-facto UI-testing solution for Xamarin apps and a great testing solution for all apps even if they don't leverage the Xamarin development platform. Microsoft will continue to support and develop the sub-components that Xamarin.UITest depends on. The deprecation of Calabash has no implications for the viability of Xamarin.UITest as a testing option. |
| 39 | + |
| 40 | +## The Tools Landscape beyond Calabash |
| 41 | +The landscape of good mobile UI & app testing frameworks has expanded over the last few years. In addition to Calabash, there are 6 major app testing frameworks available for iOS and Android: |
| 42 | +* **Espresso** |
| 43 | +* **XCUITest** |
| 44 | +* **Xamarin.UITest** |
| 45 | +* **Appium** |
| 46 | +* **EarlGrey** |
| 47 | +* **KIF** |
| 48 | + |
| 49 | +Below you can find a brief description of the tools, a tabular overview and finally some general advice to help inform your test framework and tooling selection. |
| 50 | + |
| 51 | +>Note: this guide was written in the fall of 2017. It still provides a valid overview, but the space may have changed, and more data would be available. For example, in the React Native community, the [detox testing framework](https://github.com/wix/detox) has been gaining popularity. |
| 52 | +
|
| 53 | +#### Espresso |
| 54 | +Espresso is an Android-only UI testing framework supported by Google. It's fast, based on Java, integrated in Android Studio and has a Test Recorder which aids in getting started. Similar to Calabash, it's based on the Android instrumentation framework. Since it's Android-only and Java-based there is no support for cross-platform testing. |
| 55 | + |
| 56 | +The natural target audience is Android native developers. |
| 57 | + |
| 58 | +#### XCUITest |
| 59 | +XCUITest (or UI Testing with XCTest and Xcode) is Apple's official UI Testing framework for iOS. It was announced with iOS 9 and supports iOS 9 and higher. Tests can be written in Swift or Objective-C. |
| 60 | + |
| 61 | +Similar to Apple's UIAutomation is based on the iOS accessibility system, e.g., identifying buttons via accessibility traits, identifiers and labels. XCUITest is integrated in Xcode and has a Test Recorder. |
| 62 | + |
| 63 | +XCUITest is a powerful framework and is much easier to use than the predecessor UIAutomation, but still suffers a bit from lack of maturity. It's is, however, being actively updated and improved. |
| 64 | +Since it's iOS-only and based on Swift/Objective-C there is no support for cross-platform testing. |
| 65 | + |
| 66 | +The natural target audience is iOS native developers. |
| 67 | + |
| 68 | +#### Xamarin.UITest |
| 69 | +Xamarin.UITest is a cross-platform UI testing framework for iOS and Android. It is based on the same software-infrastructure as Calabash (the Calabash servers and DeviceAgent). |
| 70 | +It's important to note that even though Calabash is deprecated, Xamarin.UITest will continue to be fully supported and developed by the Xamarin team at Microsoft. Just like Calabash, Xamarin.UITest works with any app regardless of what technology was used to build it (i.e. Xcode/Swift/Objective-C, Android/Java, Cordova, Xamarin are all supported). |
| 71 | + |
| 72 | +Xamarin.UITest has the same feature-set as Calabash, and is just as easy to use. It also has added benefits of auto-completion in IDEs due to the strong typing of C#, and supports automatically injecting the Calabash server component. There is also IDE support for Xamarin.UITest via templates in Visual Studio and Visual Studio for Mac. |
| 73 | +The natural target audience is Xamarin platform developers. |
| 74 | +#### Appium |
| 75 | +Appium is a cross-platform UI testing framework for iOS, Android, and Windows. Appium uses a client-server architecture which enables use of multiple "client" languages and test frameworks to author tests. For each supported language there is an Appium client library which is used to write tests in the programming language. The API architecture and APIs are based on Selenium: the popular open-source browser testing tool. |
| 76 | + |
| 77 | +While Appium does have a test recorder, it does not appear to be actively maintained. There are no specific IDE integrations for Appium. There is an inspector tool which helps the development process productivity. |
| 78 | + |
| 79 | +The natural target audience for Appium is test engineers which have experience writing tests using the Selenium API. |
| 80 | + |
| 81 | +#### EarlGrey |
| 82 | +EarlGrey can be considered Google's attempt to write "an Espresso for iOS". With EarlGrey, tests are written in Objective-C or Swift and run via Xcode using XCTest. Similarly to Calabash on iOS, EarlGrey uses an in-process framework which requires linking a framework into the application under test. |
| 83 | + |
| 84 | +Since it's iOS-only and based on Swift/Objective-C there is no cross-platform testing options. |
| 85 | + |
| 86 | +The natural target audience is iOS developers, particularly for teams that use Espresso on Android and want something similar on iOS. EarlGrey 2.0 is coming soon and, like Calabash and Xamarin.UITest, will leverage XCUITest. |
| 87 | +#### KIF |
| 88 | +KIF, is architecturally very similar to EarlGrey in that it uses an in-process framework which requires linking a framework into the application under test. Since it's iOS-only and based on Swift/Objective-C there is no cross-platform testing options. |
| 89 | +The natural target audience is iOS developers, particularly for teams that use Espresso on Android and want something similar on iOS. |
| 90 | + |
| 91 | +Below you can find a tabular overview of these tools in terms of features and traits: Platform support, Programming languages, Open source/proprietary, Stewardship, BDD runner support. |
| 92 | + |
| 93 | +| | Espresso | XCUITest | Xamarin.UITest | Appium | EarlGrey | |
| 94 | +| ------------- |:-------------:| :-----:|:-------------:| :-------------:|:-------------:| |
| 95 | +| **Platforms** | Android | iOS | Android, iOS | Android, iOS (Windows) | iOS | |
| 96 | +| **Languages** | Java & JVM based | Objective-C, Switf | C# | Java, Javascript, Ruby, Python (and more) | Objective-C & Swift | |
| 97 | +| **Open Source** | Yes | No | No | Yes | Yes | |
| 98 | +| **Steward** | Google | Apple | Microsoft | jQuery Foundation | Google | |
| 99 | +| **BDD Options** | [Cucumber](https://github.com/sebaslogen/espresso-cucumber) [JVM](https://github.com/sebaslogen/espresso-cucumber) | [XCFit](https://shashikant86.github.io/XCFit/) [Cucumberish](https://github.com/Ahmed-Ali/Cucumberish) | SpecFlow | Yes (depends on client test framework) | Unknown | |
| 100 | + |
| 101 | +## Recommendations |
| 102 | +We recommend asking yourself the questions: |
| 103 | +* **Dev vs QA:** Do we want developers to write tests, or do we have a QA engineering team writing tests? |
| 104 | +* **Cross-Platform:** Do we want to write cross-platform tests, sharing Test logic across iOS and Android platforms, or do we want more autonomy for iOS and Android teams. |
| 105 | +* **Languages:** Do we have strict requirements for supported programming languages and test frameworks? |
| 106 | +* **Selenium:** Can prior Selenium/WebDriver knowledge be leveraged to ease migration? |
| 107 | +* **BDD:** Do we want to use Behavior Driven Development (BDD)? |
| 108 | +* **Calabash:** are you using the more advanced features in Calabash like backdoors, invoking methods on objects, keychain interactions, and spoofing the device location, intent mocking, … This would also be technically supported by Espresso/EarlGray, however not through their API. |
| 109 | + |
| 110 | +#### When to migrate to Xamarin.UITest |
| 111 | +[Xamarin.UITest](~/test-cloud/uitest/index.md), being based on the same software infrastructure as Calabash, is very mature. It's a good fit for Calabash users, who are not intimately linked to the Ruby programming language. Xamarin.UITest also works well with SpecFlow which provides a way to leverage the '.feature' specifications in a Calabash project. (Xamarin.UITest also works with just NUnit so BDD-style testing is not a requirement.). |
| 112 | +* **Dev vs QA:** For Xamarin developers, it's an excellent choice, but basic usage of C# is enough. Can work for both Dev and QA, assuming the team is comfortable with C#. |
| 113 | +* **Cross-Platform:** Supported. |
| 114 | +* **Languages:** C# |
| 115 | +* **Selenium:** No. |
| 116 | +* **BDD:** Optional, but fully supported with SpecFlow |
| 117 | +* **Calabash:** If you like and leverage the Calabash feature set (e.g. backdoors, invoking methods, keychain, ...), Xamarin.UITest supports them. |
| 118 | + |
| 119 | +#### When to migrate to Appium |
| 120 | +[Appium](http://appium.io/), is a strong open-source player which is quite mature. Being based on the Selenium/WebDriver APIs and architecture and supporting the same programming languages, it attaches to the large set of Selenium QA engineers and the Selenium ecosystem. Appium is unopinionated about programming language and test runner. However, on-boarding is hard, particularly if you're not already familiar with Selenium. Recently Appium has transitioned away from SauceLabs and onto the jQuery foundation. |
| 121 | +* **Dev vs QA:** Best suited for QA teams as mobile app developers, unless already familiar and comfortable with the Selenium API tend to feel that it's unproductive and hard to learn (anecdotally). |
| 122 | +* **Cross-Platform:** Supported. |
| 123 | +* **Languages:** Most every language is supported, for example, Java, Ruby, Python and JavaScript. |
| 124 | +* **Selenium:** Yes. |
| 125 | +* **BDD:** Optional, depends of choice of language and framework. |
| 126 | +* **Calabash:** If you're not leveraging the advanced Calabash feature set (e.g. backdoors, invoking methods, keychain, etc). |
| 127 | +Appium provides flexibility and standardization, but it might be hard to engage developers. Also be cautious about the future of the project (e.g. timely support for new iOS and Android releases). |
| 128 | + |
| 129 | +#### When to migrate to Espresso |
| 130 | +[Espresso](https://developer.android.com/training/testing/espresso/index.html) is targeted at Android native developers, who believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test. This means that Espresso is best suited for native Android developers using Java and Android studio. |
| 131 | +* **Dev vs QA:** Works very well for native Android developers, comfortable with Java, Android Studio and Android. Can work for all QA teams familiar with Java and Android. |
| 132 | +* **Cross-Platform:** No: Android only |
| 133 | +* **Languages:** Java (JVM based) |
| 134 | +* **Selenium:** No. |
| 135 | +* **BDD:** Optional, 3rd party open source: via Cucumber/JVM: Espresso-cucumber |
| 136 | +* **Calabash:** Given the architectural similarity to Calabash, it should be possible with some work to leverage advanced Calabash Android features. |
| 137 | + |
| 138 | + |
| 139 | +#### When to migrate to XCUITest |
| 140 | +[XCUITest](https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/09-ui_testing.html) is Apple's extension of the XCTest framework to UI testing. The UI testing features can be used for 'black-box' testing where the test author has no knowledge about the internals of the app. Since XCUITest is integrated in Xcode and supports Objective-C and Swift, it works well for developers already familiar with those programming environments. |
| 141 | +* **Dev vs QA:** Works very well for native iOS developers (Swift or Objective-C). Not many QA teams will be familiar with this environment, but can be trained. |
| 142 | +* **Cross-Platformm:** No: iOS only |
| 143 | +* **Languages:** Swift or Objective-C |
| 144 | +* **Selenium:** No. |
| 145 | +* **BDD:** Optional, 3rd party open source: XCFit and Cucumberish |
| 146 | +* **Calabash:** No easy way to do it |
| 147 | + |
| 148 | +#### When to migrate to EarlGrey or KIF |
| 149 | +[EarlGray](https://github.com/google/EarlGrey) can be described as "Espresso for iOS": it's developed by Google and uses the same philosophy as Espresso. EarlGray is also very similar to XCUITest: it's iOS-only, supports Swift and Objective-C, leverages XCTest. However, in contrast to XCUITest, EarlGray, like Calabash, requires linking a framework into the application under test. This brings some additional power, and the expense of some added complexity. EarlGray is still relatively new, and perhaps not as stable as the other frameworks. |
| 150 | + |
| 151 | +EarlGrey and KIF are very similar and we don't have the necessary knowledge to recommend one over the other. One thing we can say is: KIF is a lot older and is likely to be more mature than EarlGrey. On the otherhand, KIF does not have a big player like Google funding it. |
| 152 | + |
| 153 | +EarlGrey is inspired by Espresso and "automatically synchronizes with the UI, network requests, and various queues; but still allows you to manually implement customized timings, if needed" (from GitHub: [EarlGrey](https://github.com/google/EarlGrey)). In theory, this should make writing tests easier and test should execute faster. If considering KIF vs. EarlGrey, you should wait with you decision until EarlGrey 2.0 which is shipping in the next 2-3 months. |
| 154 | + |
| 155 | +* **Dev vs QA:** Works very well for native iOS developers (Swift or Objective-C). Not many QA teams will be familiar with this environment, but can be trained. |
| 156 | +* **Cross-Platform:** No: iOS only |
| 157 | +* **Languages:** Swift or Objective-C |
| 158 | +* **Selenium:** No. |
| 159 | +* **BDD:** Optional, 3rd party open source: [XCFit](https://shashikant86.github.io/XCFit/) and [Cucumberish](https://github.com/Ahmed-Ali/Cucumberish) |
| 160 | +* **Calabash:** Given the architectural similarity to Calabash, it should be possible with some work to leverage advanced Calabash iOS features. |
| 161 | + |
| 162 | + |
0 commit comments