Evaluation of the test coverage of the Java project on the example of Apache Ignite

I participate in the development of the open source project Apache Ignite , while working on the project it became interesting for me to evaluate the test coverage and this is what came of it.

Test coverage is the most popular metric used in assessing the quality of product testing.

This is one of the few metrics that allows you to identify areas requiring attention due to the risk of missing an error, as well as to prioritize work on the modules or components of the project.

The easiest way to get a complete assessment of the test coverage of a Java project is to use the coverage runner built into IntelliJ IDEA . It allows you to set up a collection of metrics in a couple of clicks and run tests with the subsequent generation of a report.

Testing in the Apache Ignite project

The Apache Ignite project for testing uses its own test framework implemented on the basis of JUnit 3. At the time of this writing, the core module of the project contains ~ 82 thousand tests, most of which are component and require raising a cluster from several nodes, including various JVMs, with concomitant preparation of the environment.

It should be noted that ensuring the performance of such a huge regression base is not an easy task. The community constantly monitors the state of the product and corrects errors found in the framework of the initiative " Make Teamcity Green Again ".

Designated features of the project do not allow to run all the tests at once in one JVM for the following reasons:

• possible error OutOfMemoryError;
• possible failure (crash) JVM;
• possible deadlocks;
• the inability to start the test due to not stopped the node in the previous test;
• the run will take three days on one computer.

All this makes it impossible to use IntelliJ IDEA to get a report on all project tests and requires a special approach to solving the problem.

Preparation and evaluation of test coverage

Based on the work done, the most reliable approach was chosen for the task, containing the following steps:

1. Definition of a set of test classes;
2. Execution for each test class:
   2.1. running and running a test suite of a class in a separate JVM with a watchdog timer, which terminates the stream in case of a hang or test problems;
   2.2. operations for obtaining and maintaining test coverage metrics;
   2.3. cleaning the environment upon completion of the tests;
3. Merge all metrics obtained in clause 2;
4. Generate a full report.

There are many tools designed to assess test coverage, the most popular of them are:

• JaCoCo
• Cobertura
• Clover

I will not dwell on their differences, a clear table comparing the capabilities of the tools for assessing the test coverage is presented here .

To solve the problem, the JaCoCo library was chosen in order to be able to embed the solution on TeamCity , on which the existing Apache Ignite project testing infrastructure is based. TeamCity can work out of the box with JaCoCo .

The bash script and Maven were used to automate the described algorithm. The configuration of the Jacoco Maven plugin is implemented by a separate Maven profile in pom.xml.

The configuration profile of the JaCoCo plugin is shown below and implies the division into 2 separate launches:

1. Test run with connected JaCoCo agent ( prepare-agent ) for collecting test coverage metrics. The 'runDirectory' property will be passed by the script at startup, which will allow to save the results of the runs isolated;
2. Merging the results of the run ( merge ) and generating the report ( report ).

<profile> <id>coverage</id> <properties> <argLine> -ea \ -server \ -Xms1g \ -Xmx6g \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:+AggressiveOpts \ -DIGNITE_UPDATE_NOTIFIER=false \ -DIGNITE_NO_DISCO_ORDER=true \ -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true \ -DIGNITE_QUIET=false \ -Djava.net.preferIPv4Stack=true \ </argLine> <coverage.dataFile>${runDirectory}/coverage-reports/jacoco-ut.exec</coverage.dataFile> <coverage.outputDir>${runDirectory}/jacoco-ut</coverage.outputDir> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> </configuration> <executions> <execution> <id>default-test</id> <phase>test</phase> <goals> <goal>test</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.1</version> <executions> <execution> <id>default-prepare-agent</id> <goals> <goal>prepare-agent</goal> </goals> <configuration> <destFile>${coverage.dataFile}</destFile> </configuration> </execution> <execution> <id>post-merge</id> <phase>validate</phase> <goals> <goal>merge</goal> </goals> <configuration> <fileSets> <fileSet> <directory>${basedir}</directory> <includes> <include>results/*/coverage-reports/jacoco-ut.exec</include> </includes> </fileSet> </fileSets> <destFile>merged.exe</destFile> </configuration> </execution> <execution> <id>generate-report</id> <phase>validate</phase> <goals> <goal>report</goal> </goals> <configuration> <dataFile>${basedir}/merged.exe</dataFile> <outputDirectory>${basedir}/coverage-report</outputDirectory> </configuration> </execution> </executions> </plugin> </plugins> </build> </profile> 

Below is a script that implements the steps described earlier.

 #!/bin/bash #        DEVNOTES.txt # #    : ignite/modules/core # #   : 'nohup ./coverage.sh >/dev/null 2>&1 &' SCRIPT_DIR=$(cd $(dirname "$0"); pwd) echo "***** ." echo "*****   ..." tests=() while IFS= read -r -d $'\0'; do tests+=("$REPLY") done < <(find $SCRIPT_DIR/src/test/java/org/apache/ignite -type f -name "*Test*" ! -name "*\$*" ! -name "*Abstract*" ! -name "*TestSuite*" -print0) testsCount=${#tests[@]} echo "*****   ="$testsCount idx=0 for path in ${tests[@]} do idx=$((idx+1)) echo "*****  "$idx"  "$testsCount echo "*****  : "$path filename=$(basename -- "$path") filename="${filename%.*}" echo "*****  : "$filename runDir=$SCRIPT_DIR"/results/"$filename mkdir -p $runDir if [ "$(ls -A $runDir)" ]; then continue fi echo "*****  ..." timeout 30m mvn -P surefire-fork-count-1,coverage test -Dmaven.main.skip=true -Dmaven.test.failure.ignore=true -Dtest=$filename -DfailIFNoTests=false -DrunDirectory=$runDir echo "*****  ..." pkill java done #      mvn -X -P surefire-fork-count-1,coverage validate echo "***** ." 

The run of all tests with coverage coverage took ~ 50 hours on a dedicated server: 4 vCPU, 8RAM, 50 SSD, Ubuntu x64 16.04 .

The described approach can be easily parallelized into several stands, if resources are available, which will significantly shorten the time of running and obtaining an estimate of the test coverage. After embedding this solution on TeamCity , the test coverage evaluation time should take about 2 hours.

results

According to the results of the report, the coverage of the project instructions is ~ 61%.

Covering the instructions of the main components:

• Cache - 66%
• Discovery - 57%
• Compute - 60%
• Stream - 51%
• Binary - 68%
• Transactions - 71%

After analyzing the results, it became obvious that all the hot code was covered, as well as the code for correcting typical problems. Having such a toolkit it will be possible to expand the coverage to rare and atypical situations, making the product even more reliable.

PS Full audit report .