QA engineers, functional & UI testing

This is the second blog post in a series in which we describe our many years of experience with Azimo's Android app testing. The principles, goals, achievements, and most of the principles also apply to our iOS application.

QA engineers within the team

Let's go back to , the first blog post of the series. The problem wasn't with missing unit tests, but rather the QA team being unavailable when we needed them. Why did we create hundreds of unit tests, instead of hiring a dedicated QA engineering?

Quality Assurance at Azimo: The evolution of apps

Our journey, our goals, and our motivations

Even if this were possible, it wouldn't be enough to keep them or her busy all the time. Our app was only released once per month. This was not only due to limited access to the QA team, but also because of the poor quality of our code and bugs, and the lack of process automation.

This was changed by unit tests. They improved the quality of our codebase and made it easier to build the app's architecture. We adopted the "if something's hard to test, maybe it is a bug" approach and modularized our project. We broke down large classes into smaller ones, introduced dependency injection, removed SDKs from our app's logic, and also made it easier to test.

These changes enabled us to move forward and create a full-time QA engineering position in the mobile development group. Clearer code means that changes are delivered in smaller pieces, which leads to fewer bugs. We could write functional and UI testing with better modularity and abstractions. We were able to redefine the role of QA tester to QA engineer, and we could focus on a new goal: 2 weeks release train.

Automations & Functional/UI Testing

Two things were the focus of the QA engineer's subsequent months of work:

• Test the new features that will be released with each release.
• These processes can be automated as much as possible.

Back then, one of our goals was " 80% QA manual work is automated." Manual tests were not something we wanted to do more than once a week.

Test tech stack

Our testing tech stack was built on top of publicly-available tools. Espresso was used as a testing framework, and Fastlane to build automation and pipelines. They were sufficient for our initial needs. We published some open-source projects to extend these frameworks later, after we found some holes.

• ConditionWatcher -- The solution to some of our problems using asynchronous operations. Also, the way Espresso works with them. You can read our blog post Wait for It... IdlingResource, and ConditionWatcher to learn more about our problems with its IdlingResources. This tool is featured in the Android Espresso Revealed book. We are thrilled. We are humbled.
• Fastlane plug-in to manage Android emulators -- it allowed us to move AVDs management into Fastlane pipelines. You can read more about it in our blog post Managing Android Virtual Devices During Test Session.

Wait... IdlingResource & ConditionWatcher

Android Espresso: Testing asynchronous code

medium.com

As with unit tests we don't create functional or UI tests unless there is a reason. It's not about being sexual in the engineering environment. We had to face some challenges. These are the main reasons we began building functional and user interface tests.

Unit testing is not enough

We found that our product had approximately 50-60% code coverage. However, more unit tests didn't necessarily improve the quality. Despite having fewer bugs in the code, our QA engineers found more issues with 3rd party integrations and API connections.

These problems would be solved in a perfect world if API contract testing and appropriate abstractions were used. We don't live in one.

Side note about abstractions: Sometimes it is easier to write UI test within days than spend weeks creating another abstraction over navigation, files/camera manager, permissions, and so on.

Fragmented environment

There are many screen sizes. Each manufacturer makes changes to Android OS year after year. We get new versions of the system, and we have high-end and lower-end devices. Unit tests will not reveal whether there is an intensive garbage collection, a scrolling container that has not been implemented, or if Android 4.0 is missing backward compatibility.

The easiest place to test things

It is not easy to balance speed and quality when building a startup product. To survive, sometimes you have to take shortcuts. In our case, one of those shortcuts was to test some backend features via client-side platforms. You can also automate UI tests if it is necessary to manually do so.

Training for tech excellence

Sometimes, the existing solutions can be so complicated that rewriting them completely from scratch would result in many bugs. To scale them up, we must keep refactoring. You can always use UI tests if it is too difficult to write functional or unit tests.

We tested a registration process in every country using UI tests before we created a new registration process for more than 20+ countries. We were able to ensure that all the edges of our new solution were covered thanks to their help.

For the above reasons, we spent many months adding functional and UI testing to our code. We started with the most critical functionalities.

1. Register and log in
2. Transaction, payment, and price checking
3. All other: Referrals program, document upload, settings, and others.

This was our biggest victory so far. Before functional and UI testing, QA engineers took about one day to manually test our product. Automation took us the same time but allowed us to cover more cases. Our machines never got bored, and they never stopped doing the same thing (e.g., tests for every country in 20+ countries).

Side note - We wrote another piece on the benefits of automated testing here: Automation will make your engineering team free

Automated testing can set your engineering team at ease

Automation allows engineers to concentrate on the important things and unleash their creativity.

These benefits came at a cost. We ended up running hundreds of tests, each one taking many hours due to the amount of information we had to cover. This brought new challenges that we will discuss in our next blog posts. Keep watching!