Optimising Flutter Test Execution Time: A Comprehensive Guide

Testing is a critical phase in the software development lifecycle, ensuring that your application meets its quality standards. However, as a project grows, so does its test suite, potentially leading to longer test execution times. For Flutter projects, there are effective strategies to optimise these times, ensuring efficient and rapid testing processes that fit seamlessly into your development workflow. This article explores practical approaches to cutting down Flutter test execution time and coverage optimisation.

Understanding the Challenge

Flutter’s testing framework is robust, allowing developers to write unit, widget, and integration tests to cover various aspects of their applications. However, with the expansion of test suites, developers often face increased execution times, particularly when running tests with coverage collection enabled. This not only slows down the CI/CD pipeline but can also hamper developer productivity, especially in agile environments where quick feedback loops are crucial.

Strategies for Optimisation

Parallel Test Execution/Concurrency

One of the first strategies to consider is concurrency. By running tests concurrently, you can significantly reduce the overall time it takes to run your entire test suite. Flutter supports this through the --concurrency flag in the flutter test command, allowing you to specify the number of concurrent tests processes to run.

flutter test --concurrency=4

This doesn’t mean each individual test function is executed in parallel with others within the same test file. Rather, it allows multiple test files to be run at the same time, each in its own Dart VM process. However, the actual performance gain depends on your machine’s capabilities and the nature of the tests. Note: The --concurrency flag is ignored for integration tests because they often require a more controlled environment, potentially involving real or simulated devices.

The Wrapper Test Method/Test Bundling

A more nuanced approach involves grouping your tests into a single test suite, significantly reducing the performance penalty associated with coverage collection. This method involves creating a “wrapper” test file that imports and runs all individual tests. The advantage here is the reduction in the overhead of initialising the Dart VM for each test file and the optimisation of coverage data collection.

Implementing the Wrapper Test

• Create a Wrapper Test File: Place a new Dart file in your project’s test directory. Name it to reflect its purpose, such as all_tests_wrapper.dart. The naming however, is irrelevant.
• Group Your Tests: Import your individual test files and use the group function from flutter_test to encapsulate them within the main function of your wrapper test file.

// Example: all_tests_wrapper.dart
import 'package:flutter_test/flutter_test.dart';
import 'widget_tests.dart' as widget_tests;
import 'integration_tests.dart' as integration_tests;
void main() {
  group('Widget Tests', widget_tests.main);
  group('Integration Tests', integration_tests.main);
}

• Run the Wrapper Test: Execute your tests through the wrapper to benefit from optimised test execution and coverage collection.

flutter test --coverage test/all_tests_wrapper.dart

Test Sharding

While parallel execution increases efficiency, test sharding takes it a step further by splitting your test suite into smaller segments (shards) and running each segment in parallel. This can be particularly effective in CI/CD environments where multiple executors or containers are available to run tests concurrently.

Flutter’s built-in sharding mechanism divides the test suite into multiple shards and allows each shard to be run independently. This is achieved by specifying the total number of shards and the index of the current shard being run.

Implementing Test Sharding in CI/CD

• Determine Total Shards
  Decide on the total number of shards you want to split your tests into. This could be based on the total number of tests, the structure of your project, or the resources available in your CI/CD environment.
• Modify CI/CD Pipeline
  Adjust your CI/CD pipeline to run the test command multiple times, each time with a different --shard-index, ranging from 0 to --total-shards minus one.

CI/CD Pipeline Integration (GitHub Actions Example)
When integrating test sharding into a CI/CD pipeline like GitHub Actions, you can leverage the matrix strategy to parallelise test execution across multiple runners, each running a different shard of tests as a separate job.

This configuration sets up four separate jobs, each running a portion of the tests. The --total-shards=4 flag indicates the tests are divided into four shards, and --shard-index specifies the index of the shard to run in each job. You should ensure to merge the coverage reports before using the report for your needs.

Caution: When to Use Test Sharding

While test sharding is a powerful tool for optimising test execution times in Flutter projects, it’s essential to apply it judiciously. Here are key considerations to keep in mind:

• Overhead of Initialisation
  Each test shard runs in its own separate process, requiring a separate initialisation phase. This includes starting up the Flutter test environment, which can introduce overhead, especially if the number of tests in each shard is relatively small.
• CI/CD Resource Utilization
  While sharding can parallelise test execution, each shard typically runs on its own runner or container in CI/CD environments. If your CI/CD platform has limited parallel jobs or if the setup and teardown times for each job are significant, you might not see the expected decrease in total test time. In fact, for smaller test suites, the overhead of managing multiple parallel jobs might result in longer overall execution times compared to running all tests in a single job.

When to Consider Sharding

• Large Test Suites: For projects with a substantial number of tests, where running the entire suite in a single job becomes prohibitively time-consuming.
• Sufficient CI/CD Resources: If your CI/CD platform supports a high number of parallel jobs and the overhead of setting up additional jobs is minimal.
• Balanced Shard Distribution: When you can evenly distribute tests across shards to ensure that no single shard becomes a bottleneck.

Best Practices for Implementing Test Sharding

• Evaluate Your Needs: Start by analysing your current test execution times and CI/CD resource availability. Implement sharding incrementally, beginning with a small number of shards to measure the impact on total execution time.
• Monitor and Adjust: Continuously monitor the performance of your test execution times after implementing sharding. Be prepared to adjust the number of shards based on the results and any changes in your test suite or CI/CD environment.
• Balance Test Distribution: Aim for an even distribution of tests across shards to avoid imbalances that could lead to some shards finishing much earlier than others, thereby not fully utilising parallel execution.

Test sharding in Flutter can be a double-edged sword. While it offers the potential for significantly reduced test execution times, especially for large and complex projects, it can also introduce overhead that may outweigh the benefits for smaller projects or in resource-constrained CI/CD environments. Careful consideration and continuous monitoring are key to leveraging test sharding effectively.

Very Good Cli

The Very Good CLI, developed by Very Good Ventures, introduces an innovative approach to optimising Flutter test execution. The CLI’s very_good test command supercharges Dart and Flutter apps testing with performance optimisations and developer experience improvements.

Integrating Very Good CLI into your development workflow is straightforward. Install it globally via Dart, and use the very_good test command to run your tests. The command’s flags and arguments, such as — coverage, — recursive, and — min-coverage, provide granular control over how your tests are executed and how coverage is collected.

# Install Very Good CLI dart 
pub global activate very_good_cli 

# Run optimized tests with Very Good CLI 
very_good test --coverage

CI/CD Integration

Integrating these strategies into your CI/CD pipeline can further enhance efficiency. Use CI/CD capabilities to leverage parallel execution and test sharding, and consider the wrapper test method for large, complex projects that suffer from significant delays in test execution.

Benchmarking and Comparison

To truly appreciate the impact of these optimisation strategies on Flutter test execution times, it’s beneficial to conduct a benchmarking exercise. This section compares regular flutter test execution times against optimised runs using the strategies discussed above, including running tests with coverage.

Benchmark Setup

To ensure a fair comparison, tests should be run under similar conditions and on the same hardware. For the purpose of this benchmark, we’ll consider some Flutter project with a mix of unit and widget tests distributed across several features.

Running Standard Flutter Tests

First, run the entire test suite using the standard flutter test command:

time flutter test

Then, run the tests again with coverage enabled:

time flutter test --coverage

Record the total execution time for each run.

flutter test on local machine unoptimised

flutter test on github actions unoptimised

Running Optimised Tests

Next, apply the optimisation strategies:

• Parallel Test Execution: Use the --concurrency flag with an appropriate value based on your machine's CPU cores.
• The Wrapper Test Method: Combine all tests into a single suite using a wrapper test file.

Run the optimised tests and the optimised tests with coverage, recording the execution time for each:

# For The Wrapper Test Method time 
flutter test --coverage test/all_tests_wrapper.dart

flutter test on local machine optimized

flutter test on github actions optimised

Results and Comparison

After running the tests using both standard and optimised methods, we compare the execution times. Its observed that there is significant reductions in total test runtime, especially for the optimised tests with coverage. The exact performance gains will depend on factors like the number and complexity of tests, project size, and your CI/CD environment’s capabilities.

Optimised Tests with Coverage vs. Flutter Test with Coverage

When comparing optimised tests with coverage against standard flutter test runs with coverage, the difference can be stark. Coverage collection often adds considerable overhead, but by aggregating tests and minimising the initialisation overhead, the optimised approach can drastically reduce this penalty, resulting in faster CI/CD pipelines and more efficient development cycles.

Conclusion

Optimising test execution time in Flutter projects is essential for maintaining a fast, efficient development pipeline. Through parallel execution, test sharding, and innovative approaches like the wrapper test method, developers can significantly reduce testing times. This not only accelerates the CI/CD process but also improves developer productivity by providing quicker feedback on the quality of the codebase. Implementing these strategies ensures that your Flutter project remains agile, robust, and ready for rapid iteration.

The goal of optimisation is to find the sweet spot where the benefits of reduced execution time outweigh the costs of increased complexity and resource usage.

Happy testing 🧪

Resources

• flutter/flutter issue
• Very good cli
• Supercharge your Flutter tests with Very Good CLI

Originally published at https://dev.to on April 10, 2024.