Easy A/B testing on Android with AndroidX Navigation Component
Creating a successful app requires taking data-driven decisions. A great tool to gather valuable insights from your users is A/B Testing. There are a lot of resources online explaining the theory behind it (like here) but essentially is deploying 2 or more different variations of one feature to different sets of users and comparing how these variations perform against predefined goals (like increasing day one retention, conversion rate, etc…) As a result, you will have (hopefully) one winning variation, which will outperform the others, and which will become your new standard for all users (of course the other variations will be gone for good…)
On the technical side, when A/B Testing, the main challenges are not to pollute your business logic and/or the main codebase with test-specific code and design a solution that will make the necessary after test cleanup easy leaving the resulting code in a good shape, like no test ever happened there.
Navigation challenges
Google released the AndroidX Navigation component library in 2018 as a part of Jetpack. It favorited one activity — multiple fragments app architecture, hiding most of the weird/bad lifecycle related issues fragments were notorious of. It’s powerful and at the same time really easy to work with, providing a nice graphical navigation editor and making fragments-scary developers feel much more comfortable using them. If you haven’t tried it yet you can read more about it here and take a look at the codelab here.
Another great use case we found for navigation component, is for (you guessed right) A/B Testing and has been proved very successful helping with the technical challenges described.
Consider the scenario, that you want to test a feature (for example user onboarding) and you have 3 different variations of it, with 4 screens in each variation. Baseline, Variation 1 with completely different screens and Variation 2 with a mix of new and Variation 1 screens in a different order.
It seems like a nightmare designing the navigation logic, with ifs and whens all over the place and using a ViewPager or just the fragment manager, manually handling all the fragment transactions. But with the navigation component, all of these issues do not even exist!
To demonstrate the new approach using the navigation component I created a new toy project, the full source is here:
Navigation Component to the rescue
The main idea is that there should be 3 different navigation graphs, one for each variation of the AB Test (Baseline, Variation 1, Variation 2). Each navigation graph will contain 4 destinations (according to the test scenario we described before, but it can be any number and not even the same between variations) which are fragments, and 3 actions connecting the destinations. To help visualize, I took screenshots from the android studio navigation editor. The destination fragments are the colored boxes, and the actions are the arrows.
Each different (some are shared between variations) fragment has a different class and XML layout file. In a real-life scenario, this works really well for cleanup purposes since it’s an A/B Test at the end of which we will just delete all the fragment classes and layout files that didn’t make it, without worrying about refactoring for the winner variation.
The A/B tested feature’s hosting activity (ABTestActivity in the project) needs to know somehow which navigation graph to inflate. In a big project, we would like to inject this info but for simplicity now, there is an enum class ABTestVariation, and the appropriate variation is passed with the activity’s launching intent:
This approach is very straightforward and looks also clean, but there is a big challenge. In order to navigate between fragments, you have to specify either the destination you want to navigate to or the action you should take and both of them require an id:
findNavController().navigate(R.id.action_b_to_a)and since we have 4 different destinations and 3 different actions in 3 different XML navigation graphs, how would we know which id to use in the activity which handles all of them? It looks like we are trapped again with whens and ifs, or the fragments should somehow take this info and pass it back into the parent activity which is also not great.
Finally, after a lot of trial and error, trying multiple approaches, came the Eureka moment! The ids of the destinations in the different graphs don’t have to be different and… the ids of the actions in even the same nav graph don’t have to be different! And that means we can name each action with id next and then the activity will try to navigate to… next!! And if this action doesn’t exist, well that means that we reached the end of the graph! That’s so easy!
Cleaning up after the test is also super easy, just delete the fragments, their layouts and the nav graphs (and if you want, make the activity inflate the same graph without passing this info with the intent).
Happy A/B Testing!
