At 3SS we work on many front end and client-application projects for various platforms and devices as part of TV and VOD solutions. Compared to “traditional apps” these solutions are far more complex and involve many more features when it comes to testing and maintenance. This is because the increased volumes and variety of data and use-cases result in a large number of scenarios that require testing and validation. Moreover, quality of experience in a video-driven environment is of the highest importance since the interruption of a stream, buffering or slow reaction times can have a devastating impact on user satisfaction. This is why it is essential to ensure that software we build satisfies the very highest standards.
With classic manual testing and regression practices it is impossible to anticipate and cover 100% of all the different possible scenarios and regularly re-test them as part of ongoing development and maintenance over several months or years.
This is why automated testing on different levels of a system has become such an important factor to ensure quality while optimizing the effort and resources required.
There are numerous ways of carrying out automated testing, ranging from unit-testing at development-time to UI testing, to integration and end-to-end testing until blackbox testing on the final device with automated inputs.
In this article we look a bit deeper into UI testing and how it can be automated on iOS.
You probably have already heard about the advantages of automation of UI testing –many people talk about it when speaking about software quality and about how adding automation leads to more extensive and highly granular testing of applications.
In this article we will describe 3SS’ experiences and findings in automation of UI testing for iOS: basic approach, tools, workflow and of course how it helps to improve the quality of software solutions.
Speaking about UI testing, the best practice to approach UI testing is to follow the FIRST paradigm, which is an acronym for following criteria:
- Fast – tests should run quickly;
- Independent/Isolated – tests should not do setup or teardown for one another;
- Repeatable – tests should obtain the same results every time you run them with the same input data;
- Self-validating – tests should be fully automated and the output should be “pass” or “fail”;
- Timely – ideally, tests should be written just before you write the production code they test.
Before implementing the automated UI testing solution you need to answer some key about how and what to test.
What should be tested
Regarding the first question, the recommended basic approach is the following:
- Find the required UI element on the screen;
- Interact with the app via UI elements in the same way as the user does;
- Validate changes through assertions.
In order to get the best out of UI testing it is important to keep in mind: UI-testing does not aim to provide end-to-end testing of features and integration. Rather, the goal of UI testing is to verify that the UI works as expected from a user’s perspective and to design the test-cases accordingly.
This approach can be easily transformed into Given/When/Then approach, which is a good practice for all automated tests:
- Given – set up any values needed and find needed elements;
- When – execute the code being tested, and the means to interact with elements;
- Then – assert the result you expect and check/verify.
An example of this:
Given I am on the home-page and I am logged in
When I click on the play button of the main teaser
Then the video should start in fullscreen.
Using this syntax the tests can be understood by automation tools as well as test-engineers for manual testing.
How should be tested
So, we have an answer for the first important question before starting to implement the UI tests in iOS.
Now let’s find out the answer for the second one – how to test. And we will start with tools set.
For our automated UI test solution we have used the native XCTest framework –Xcode’s built-in test framework which allows developers to write UI and unit tests in the same workspace with tested project in Xcode.
We have chosen this framework due to the following benefits it provides:
- No additional components required – everything required to write automated tests is already present in XCTest;
- Easy to learn – it is easy to start and convenient to use XCTest;
- Native iOS languages support – XCTest supports Swift and Objective-C, so you won’t need to learn additional languages;
- Xcode Test Recorder – XCTest supports record feature, which is a powerful tool for UI testing. It allows you to record the UI actions and generate the code for them in test classes;
- Easy integration with CI – Xcode allows you to run the tests from shell script and integrate them with wide CI solutions, in our case GitLab-CI
- Fast test execution – XCTest runs tests in a process separate from the tested app, which increases the speed of test execution.
Using this setup we can access UI elements based on identifiers and queries. Once you have found the required UI element, you can interact with it as the end-user does with the help of native functions, simulating user actions (tap, swipe, long tap, etc).
All that remains is to check UI changes with the help of assertions and to validate whether the actual result matches the expected result. Depending on whether the condition result satisfies the expectation in the assertion or not, the test is a pass or fail.
We generate human-readable html output with activity screenshots from test passes and fails, alongside a description of the steps taken, in order to create reports. As a result, all tests can be faithfully reproduced and further analyzed.
All steps, including the creation of these reports, are automated and executed in the background.
Remote device testing
Another challenge of UI testing is the wide diversity of devices, all with different display sizes and hardware resources. Setting up an automated testlab with dedicated hardware to accommodate all devices is complex and can be prohibitively expensive.
To overcome this, numerous providers in our industry offer cloud-testing, either on real devices based on assigned slots or by using emulators. One such provider is AWS Device Farm.
There are multiple advantages of using a cloud-testing service:
- UI tests can run on real devices which don’t need to be purchased, owned and connected in a local lab – something that is otherwise nearly impossible for Android
- The tester can select a specific batch of devices on which tests will be running;
- Enables the tester to collect videos, logs, and performance data so they can dive deep and solve problems quickly;
- Simulates real-world environment (by configuring location, language, network connection, application data, and installing prerequisites);
- Facilitates integration with your CI, enabling setup to keep the automation
Multiple aspects of UI software need to be comprehensively tested by the software tester to ensure sufficient quality and acceptance.
Automating the UI testing process saves time and other management resources for testers and developers. The main advantage of testing the UI with automation is that the same piece of functionality does not need to be re-checked manually after adding new features or fixing bugs.
It’s easy and inexpensive to run pre-coded tests after each change, so you can make even the most risky changes, even at the very end of the development cycle, and be sure that your application is still stable and sound.
Developers will find this facility useful, because bugs will become known at the development stage with clear steps available in order to reproduce.
Bugs found during the development phase are always more easily and economically fixed than those discovered during production and maintenance.
Of course creation and maintenance of UI tests necessitate additional efforts during the development phase. But over the long-term, many person-hours of testing and issues resolution are saved by automating this.
With manual testing only, the effort required to write, execute and maintain test-cases constantly increases over time as features and device-landscape grow. With automated testing, the resources, especially time, needed to meet requirements will be much reduced over time. Additionally, by just executing a subset of the tests, very speedy sanity-tests or smoke-tests following a release can be executed. This instantly prevents errors from ever being deployed, for example in combination with blue-green deployments based on a CI Flow.
3SS uses similar approaches as those outlined here for iOS on other platforms such as Android (mobile and STB) and Web.
If you are interested in learning more about this topic and exploring how 3SS can help you benefit from automated UI testing for your TV projects, feel free to contact us at …
Authors: Serhii Yanenko, Andrew Stebai, www.3ss.tv