Flutter: Choosing Mobile UI Tests Automation Tool
In the large-scale Flutter projects that our team has been working on for the last 4+ years, the topic of Mobile UI Tests Automation has always been on the table. With our first app, we released updates every month, gradually moving to biweekly releases for subsequent apps. The automation of functional application testing, conducted at least on a nightly basis, facilitates faster and more confident application releases. The importance of this is clear to both the team and the business.
Our first Flutter app used Flutter Integration Testing, in combination with Github Actions and Firebase Test Lab. However, our experience with Flutter Integration Testing was not the smoothest. Additionally, its limitations in covering WebViews or Native views could pose significant problems.
Another Flutter team used the first version of Patrol on a different project. In both instances, it was primarily the developers who wrote these tests, but there was a desire among QAs to take ownership.
Our latest Flutter projects had a more complex setup. The teams have been working on a white-label framework, building several Full Flutter applications while also integrating some Flutter features into existing native apps (iOS + Android). These existing native apps already had automation running, so last year we conducted an investigation to identify which automation tool would best suit our needs for the Full Flutter ones.
The Tools We Considered
The table below provides a brief overview of the tools under consideration. Please note that some known limitations of certain tools are not depicted in the table.
What I found really interesting is that Maestro was formerly known as Conductor, and Patrol was previously named Maestro! You can check this rename in the Maestro repo and this one in the Patrol repo.
It’s important to note that black-box testing of Flutter views is conducted based on the provided semantics information. By default, this information is included in widgets that display text. However, you can attach semantics information to any widget using Flutter’s Semantics widget.
Additionally, the long-living accessibility-id issue was finally resolved in Flutter 3.19, thanks to Bartek Pacia! As a result, a new accessibility identifier property of the Semantics widget is now propagated to the native accessibility hierarchy.
What Was Important to Us?
We paid special attention to the following aspects of the tools under consideration:
- Maturity and maintainability
- Black-box testing
- Inspecting tools
- Compatibility with our QAs' skills
Our Choice
Given that WebdriverIO is a mature tool already in use for our native apps and familiar to our QAs, and since it is technically capable of covering both full Flutter and Add-to-app Flutter apps, our decision to broaden its use was straightforward. This choice also amplified our motivation to pay closer attention to the Semantics trees in our apps, as they are not only crucial for impaired individuals, but also serve as a bridge to UI Tests automation tools!
Bonus: A Demo of Maestro’s Ease of Use!
Even though we didn’t choose Maestro for our latest Flutter apps, I was struck by its speed and ease of use.
So, to give you a notion of it, I wiped off the dust from my first Flutter pet project I started 5 years ago and decided to automate a test case. The project is MemMe, a laconic flashcards app designed to make memorization easy.
With just one command, I installed Maestro and began experimenting with Maestro Studio. I wrote the following flow to test the case of adding a new flashcard to the freshly installed application. It was so quick and enjoyable to write that I finished the test before even finishing my cup of tea!
The animation below demonstrates the test flow run. By the way, the video was recorded using the built-in feature in Maestro.
Conclusion
Don’t hesitate to invest time exploring the world of Mobile UI Tests automation. With the right tools and understanding, this investment can significantly enhance your ability to deliver high-quality, reliable applications. It’s an investment that quite often pays off.
