-
Notifications
You must be signed in to change notification settings - Fork 22
Test Strategy
Please see https://docs.google.com/document/d/1r0h2QFhLTk56OmPKEtJ6TFQfVdYKHe2WHck6Rx3ulSI/edit# for the latest test strategy.
This is currently manual in focus and a living document.
The Pride Process Flow Chart shows the general flow of testing in the context of the entire process. To be used as a guide only, there will of course be exceptions that aren’t accounted for.
When you pick up a pull request for testing, please change the label from “ready for test” to “test in progress”.
- Pull the pull request
- If you have access to physical devices then always prioritise testing on these
- If not, then use BrowserStack if you have access.
- Otherwise you can use Android Studio / Xcode Simulator (or another emulator of your choice)
- Find an issue? Investigate, then raise it.
- If you are happy that there are no issues and that the work meets the acceptance criteria and spec of the Trello ticket, change the label to “qa approved”.
- Merge and close the pull request.
- Once the pull request is merged and the app rebuilds successfully, smoke test the changes on the master branch to ensure integration has gone smoothly.
You can use either Hockeyapp or Fabric to get access to Alpha/Beta builds for Android and iOS. Request access to these services in the Pride Slack channel.
See the Downloading Builds wiki page for more information.
BrowserStack is a useful testing tool because it allows you to remotely test on physical devices.
If you don’t have access, create an account and request access in the Pride slack channel or message me (declan.slevin) directly. Please note that the Pride in London organisation is separate from the Red Badger account and that your email address can only be associated with one organisation at a time. Therefore if you need to maintain access to the Red Badger organisation you should create a new account for use with Pride.
To test the app with BrowserStack, the most straightforward way is upload an .apk or .ipa:
- Log in to BrowserStack, go to the App Live page.
- Drag and drop the .apk / .ipa installer onto the browser window.
- Then you can choose which device you want to test the app with.
Getting an .apk / .ipa:
- .ipa
- Get access to Hockeyapp and an iOS device registered
- Log in to the Hockeyapp website
- On Dashboard > Apps, choose ‘iPhone | alpha’ or ‘iPhone | beta’
- Click the Download button
- .apk
- Either:
- Open the Hockeyapp website Dashboard > Apps
- Choose ‘Android | alpha’ or ‘Android | beta’
- Click the Download button
- OR:
- Open the Android Studio project
- Build > Build APK(s)
- Either:
You can emulate different Android devices and OS versions using Android Studio. Once you have installed it and have the application running, you will need to install the required SDKs:
- Open SDK manager (icon near the right of the top toolbar)
- Enable your desired SDK versions
- Click Apply and OK, then accept the terms and install
Once you have your SDKs installed, open the ‘AVD Manager’ (icon near the right of the top toolbar) and create your device. NOTE: only Google devices are included by default.
https://developer.android.com/studio/run/managing-avds.html
You can create custom devices (i.e. Samsung) by manually entering the device specs. It is even possible to add the device skins: http://developer.samsung.com/galaxy/emulator-skin/guide
You can find Samsung device specs here: https://www.devicespecifications.com/en/brand/d1cd1
Run the packaging server in a terminal with yarn run-android
With the android project open, click the “Run app” icon (Ctrl + R) and select your device from the list under “Available Virtual Devices”.
Xcode makes things a little simpler than Android Studio to emulate hardware devices (and it comes with most iOS hardware profiles included by default). To do so, open the xcode project (/pride-london-app/ios/PrideLondonApp.xcodeproj). From the top toolbar, select your desired device, then click the run icon.
There is a guide for raising issues here : https://github.com/redbadger/pride-london-app/blob/master/CONTRIBUTING.md
If you encounter what you believe to be an issue in the course of testing, please do the following:
- Inform the developer responsible for the pull request you are testing.
- If the issue is serious enough, they may wish to fix it before the pull request is merged. In this case, do not raise an issue - instead, change the label on the pull request to “awaiting fixes”.
- Unless told otherwise, raise an issue on GitHub. If relevant, comment on the Trello ticket with a link to the GitHub issue. You may also consider the following:
- Test the master branch and see if the issue exists there also.
- If you have access to multiple devices/platforms, attempt to reproduce the issue to determine whether it is an isolated, platform/version specific issue.
Some labels have been set up in GitHub to categorise bugs or act as a signifier that action is required. The relevant ones for testers are:
- bug - self-explanatory, if an issue is considered a bug
- enhancement - issue is not strictly a bug but would enhance quality/user experience
- accessibility - issue is specific/relevant to accessibility
- uxd - issue requires the attention of UXD
- high priority - issue should be prioritised first
- awaiting fixes - testing has started on pull request but requires fixes from a developer
- qa approved - pull request has been fully tested but can’t yet be merged for whatever reason
Please tag logged Issues with the relevant GitHub labels, i.e. “bug”, “enhancement”, “accessibility”, etc. If you consider the issue to be a high priority, then add the “high priority” label.
It is important that users have a pleasant experience using the London Pride app regardless of any impairments they may have. Most devices come with an accessibility section in the settings which can be utilised to assist effective testing.
We want to focus on the following:
- Increase the font size in your phone settings as large as you can (at least 200%) and view the app and check that any text is affected by this change of setting and is still legible.
- Enable the text-to-voice function and tap on content in the app.
- All non-text content should have a text alternative that describes the information provided by the content, as long as it serves a purpose. If an image is purely decorative then it should not be read out.
- Images that are informative or functional should have correct alternative text e.g. an indication that this is an external link etc.
Useful guides here:
https://www.w3.org/WAI/mobile/
https://accessibility.uber.com/
https://support.google.com/accessibility/android/answer/7101858?hl=en-GB
https://developer.mozilla.org/en-US/docs/Web/Accessibility/Mobile_accessibility_checklist
Users won’t always have a reliable data/wi-fi connection, some won’t have a 4G plan and others will live in areas where data coverage is flaky. Therefore it is necessary to consider testing in such a way that simulates these conditions.
- Enable Airplane mode and observe how the app and content behaves. It should fail gracefully if attempting to load new content. An informative message as to why should be shown to the user.
- Most phones will have options to limit network speed to 2G/3G. Enable this and observe how the app and loading of content performs. Failure to load content should fail gracefully with an informative message as to why displayed.
- Pay attention to app performance - if you feel it is taking an unacceptably long time to load content or respond, please raise this with the team.