How to Provide Testwise Coverage for Test Impact Analysis (TIA)
In order to use the Test Impact analysis, you'll need to provide test coverage on a per-test basis to Teamscale. The necessary information can be described as a Testwise Coverage report.
TIA for the Java Virtual Machine (JVM)
For JVM-based languages there exists an extension to the JaCoCo coverage profiler, which can be instructed to generate Testwise Coverage.
TIA with Gradle
For Gradle based builds with JUnit Platform based tests there also exists a Gradle plugin. The plugin configures the test execution to use a custom Test Engine which handles the communication with Teamscale and delegates the actual execution to existing Test Engines.
Tutorial
We also have a Tutorial on how to set up Test Impact Analysis in a Gradle-based Java project.
TIA with Maven and JUnit 5
If you use JUnit 5 (or any test framework built on JUnit 5, e.g., Serenity) for your tests, you can use our teamscale-maven-plugin to easily run TIA in your builds. It configures Surefire and Failsafe, so they fetch the list of impacted tests from Teamscale, execute them, and record and upload Testwise Coverage. This also works for Spring Boot applications.
We recommend that for regular, fast feedback, you run only impacted tests for most of your builds. It is, however, necessary to also run all tests from time to time (e.g., in a nightly or weekly build). This allows Teamscale to regularly update Testwise Coverage information for all tests. To achieve this, you can use Maven profiles and enable the teamscale-maven-plugin
only in a special profile.
<profiles>
<profile>
<id>tia</id>
<activation>
<property>
<name>tia</name>
</property>
</activation>
<dependencies>
<dependency>
<groupId>com.teamscale</groupId>
<artifactId>impacted-test-engine</artifactId>
<version>30.0.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>com.teamscale</groupId>
<artifactId>teamscale-maven-plugin</artifactId>
<version>30.0.0</version>
<executions>
<execution>
<goals>
<goal>prepare-tia-unit-test</goal> <!-- Enables TIA for Surefire unit tests -->
<goal>prepare-tia-integration-test
</goal> <!-- Enables TIA for Failsafe integration tests -->
</goals>
</execution>
</executions>
<configuration>
<teamscaleUrl>http://localhost:8080</teamscaleUrl>
<projectId>teamscale-project-id</projectId>
<username>build</username>
<accessToken>${env.TEAMSCALE_ACCESS_TOKEN}</accessToken>
<includes>
<include>*com.your.company.*</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
List of All Available Parameters
To list all available parameters run mvn com.teamscale:teamscale-maven-plugin:help -Ddetail=true
(also works without a maven project).
This adds the plugin and configures the connection to Teamscale. We recommend you also set at least one include
pattern so coverage is only recorded for your code, not for 3rd party libraries.
It also configures Surefire and Failsafe to use our impacted-test-engine
. This engine automatically fetches impacted tests from Teamscale and executes them. Make sure the engine version matches the teamscale-maven-plugin
version.
Parallel Tests Not Supported
Parallelized tests are not supported by this plugin. When tests run in parallel, it is impossible to record Testwise Coverage as the coverage from the tests running in parallel cannot be distinguished anymore. Make sure to set <forkCount>1</forkCount>
and <threadCount>1</threadCount>
(c.f. the Maven documentation) for all tests where you enable the provider, otherwise a TIA-enabled build will fail. These settings are the defaults for Surefire and Failsafe.
Use of <reuseForks>true</reuseForks>
It is strongly discouraged to set<reuseForks>true</reuseForks>
as this will start and close a new JVM for each test case. Since the Teamscale-Maven-Plugin writes a coverage report every time the JVM closes, this can lead to performance problems on some systems.
The plugin sets the right system property to attach itself to Surefire and Failsafe tests. Surefire and Failsafe pick up the system property automatically and apply it to any JVMs they spawn.
Spring Boot
The plugin also works out-of-the-box for Spring Boot applications. Here, the Spring Boot Maven plugin will pick up the JVM options and pass them to the Spring Boot application that is started before the integration tests.
In some cases, your system-under-test is not launched by these plugins but by some other means (e.g., another plugin). In this case, make sure the argLine
property is passed to the JVM of your system-under-test. If you do not want Surefire or Failsafe to pick up the argLine
automatically or want to add additional parameters, you can change the name of the property that holds the JVM arguments by setting the configuration property <propertyName>
.
If you need to add additional parameters to the Teamscale JaCoCo agent, you can use <additionalAgentOptions>
.
Troubleshooting
The plugin creates a log file in target/tia/agent.log
. You can enable debug logging by adding the <debugLogging>true</debugLogging>
configuration property.
If the log file is not created when executing your tests, check if you have <argLine>...</argLine>
in your pom.xml
and remove it. This overwrites the arguments we set in the plugin and thus de-registers the coverage profiler.
In some cases, the test reports generated by Surefire/Failsafe may also contain error information. You can find them under target/surefire-reports
after all tests have finished. Check the stdout/stderr of failing tests for additional error messages.
If the build fails after applying the above POM changes but works without them, try to re-run the build with -e
to discover the error that caused this.
If you notice largely increased test runtimes on systems that set <reuseForks>true</reuseForks>
it is recommended to set the <tiaMode>exec-file</tiaMode>
. Writing a testwise coverage report every time the JVM closes can be time-consuming, so writing raw .exec files instead, has been shown to increase test execution times. There is a separate Maven goal com.teamscale:teamscale-maven-plugin:testwise-coverage-converter
that can then be run to batch convert the created raw .exec files into a testwise coverage report. You can use the com.teamscale:teamscale-maven-plugin:upload-coverage
goal to then upload the created testwise coverage report to Teamscale.