Test coverage explains with lcov on Flutter

Introduction

As developers, we want to have a clear view of how many covers our code source has. Somethings we want to test only a part of our codebase or whatsoever, and it can be really painful to do that.

With Flutter, we can make test coverage to see how many lines of code we cover. But something we don't want to test everything in our codebase. Imagine a Clean Architecture with data/domain/presentation, and we want to cover only data and domain, we will see in this article how to do it.

Tools we will use:

• Flutter => install here
• lcov + genhtml => install here
• Some knowledge about test and clean architecture in Flutter (see this post create by Reso Coder for more info, you can also check my previous article about dart test here)

The coverage with flutter test

Before creating some scripts to handle the coverage for us easily I want to make a quick re-brief of creating a coverage with flutter.

The command you need to know is:

flutter test --coverage

This will create a folder named coverage on your root project, and put a file named lcov.info. This is a standard output for test coverage in many other languages.

What is good with this file is we can manipulate it to remove some files for example. Like so:

lcov --remove coverage/lcov.info 'lib/main.dart' -o coverage/lcov.info

Here we remove the main.dart from the test coverage.

Ok having this .info is nice but how we can look into which line of code is not cover? Thanks to the command genhtml we can create a little website (in local) to see the test result with a nice dashboard. Take a look:

genhtml coverage/lcov.info -o coverage

This will create all the files necessary for the dashboard. Then we just have to open the index.html like so:

open coverage/index.html

And we will see a nice dashboard like this:

On the top right we can see the global coverage and you can also click on a particular folder to see a file for example the article.dart:

Here we can see that this file is not fully tested. We miss the toJson() method. We can resolve that easily with a test:

  test(
    'should return a JSON map with the proper data when call toJson',
    () async {
      // arrange
      final result = tArticle.toJson();
      // act
      final expectedJsonMap = {
        'title': 'My great article',
      };
      // assert
      expect(result, expectedJsonMap);
    },
  );

And now if we re-create the coverage and open it again we should saw:

Scripts

Now we have a clear view of what is a coverage test and a way to see each line of code that is not tested, but what's happen in a real application like a clean architecture with data/domain/presentation and we don't want to test the presentation? We will see how to do it. But first, let's make some scripts to simplicity our life and prepare the field for a future CI.

Import file coverage

First, we need to talk about an issue on Flutter that forces us to create a file to import all our .dart. I will not talk about it in detail but this is the script:

#!/bin/sh
file=test/coverage_helper_test.dart
printf "// Helper file to make coverage work for all dart files\n" > $file
printf "// **************************************************************************\n" >> $file
printf "// Because of this: https://github.com/flutter/flutter/issues/27997#issue-410722816\n" >> $file
printf "// DO NOT EDIT THIS FILE USE: sh scripts/import_files_coverage.sh YOUR_PACKAGE_NAME\n" >> $file
printf "// **************************************************************************\n" >> $file
printf "\n" >> $file
printf "// ignore_for_file: unused_import\n" >> $file
find lib -type f \( -iname "*.dart" ! -iname "*.g.dart" ! -iname "*.freezed.dart" ! -iname "generated_plugin_registrant.dart" \) | cut -c4- | awk -v package="$1" '{printf "import '\''package:%s%s'\'';\n", package, $1}' >> $file
printf "\nvoid main(){}" >> $file

The most important in this script is the package name. For example, if I want to use this script for this project I will do something like:

sh scripts/import_files_coverage.sh lcov_test

And it's will generate a file in test/ named coverage_helper_test.dart.

Create clean lcov and generate html

Now we can create the second script that will handle all the test/coverage and the generation of the html dashboard.

#!/bin/bash
set -e
flutter pub get
sh scripts/import_files_coverage.sh lcov_test
flutter test --coverage
lcov --remove coverage/lcov.info \
'lib/main.dart' \
'lib/*/*.g.dart' \
'lib/features/*/presentation/pages/*.dart' \
'lib/features/*/presentation/widgets/*.dart' \
-o coverage/lcov.info
if [ -n "$1" ]
then
    if [ "$1" = true ]
    then
        genhtml coverage/lcov.info -o coverage
        open coverage/index.html
    fi
fi

You can see in line 4 we call our first script before the coverage, then we remove some files that we don't want to test, every .g.dart because it's a generated file and all files in pages and widgets from every feature. With that, we test only the business logic data/domain.

At the end of the script, you can see a condition if we pass an argument to the script to generate/open the index.html. It's helpful when we use this script in a CI.

IMPORTANT: If you want to remove files from the test coverage add them to this script.

And now we can run the script like so:

sh scripts/create_clean_lcov_and_generate_html.sh true

And you should see the test and the dashboard after it. It's will be really helpful when we are on TDD mode.

Tips: Create an alias for this script if you use it a lot:

alias ftest="sh scripts/create_clean_lcov_and_generate_html.sh true"

You can also add the coverage_helper_test.dart and the coverage folder to your .gitignore because it's not something we want to add/push every time:

# Coverage test result
/coverage/
/test/coverage_helper_test.dart

Github Action for CI

I will use GitHub action to illustrate how to use this script in a CI, of course, you can use this in other CI.

First, we need a script to stop the CI if a % of the cover is not ok:

#!/bin/bash
set -e
PATH_COVERAGE=$1
# Change this variable to increase or downgrade the min coverage percentage:
MIN_COVERAGE_PERC=100

percentageRate=$(lcov --summary "$PATH_COVERAGE" | grep "lines......" | cut -d ' ' -f 4 | cut -d '%' -f 1)

RED='\033[0;31m'
GREEN='\033[0;32m'

if [ "$(echo "${percentageRate} < $MIN_COVERAGE_PERC" | bc)" -eq 1 ]; then
    printf "${RED}Error: Your coverage rate is to low, expected ${MIN_COVERAGE_PERC}% but have ${percentageRate}%.\n"
    printf "${RED}Please add more tests to cover your code.\n"
    printf "${RED}To see in local your coverage rate use:\n"
    printf "${RED}    sh scripts/create_clean_lcov_and_generate_html.sh true\n"
    exit 1
else
    printf "${GREEN}Coverage rate is fine 👍. Build continue...\n"
fi

This will stop the CI if the coverage is lower than 100% (MIN_COVERAGE_PERC).

Now we have to create the GitHub action workflow:

on: pull_request
name: Flutter test coverage
jobs:
  tests:
    runs-on: ubuntu-20.04
    steps:
      - uses: subosito/flutter-action@v1
        with:
          flutter-version: '2.2.2'
          channel: 'stable'
      - name: Tests
        shell: bash
        run: |
          sudo apt-get install lcov
          sh scripts/create_clean_lcov_and_generate_html.sh false
          sh .github/scripts/get_coverage_percentage.sh coverage/lcov.info

It's a simple workflow, that tells us that on every pull request this will run and create the coverage file then check if the % is at least MIN_COVERAGE_PERC, if not it will fail the PR.

Conclusion

You can now have a better understanding of test coverage and even customize your result and show it in a nice dashboard. Hope you enjoyed the tutorial. See you for the next article!

Like always you can find all the code associate to this tutorial on my repo: