How to Provide Testwise Coverage for Test Impact Analysis (TIA)

In order to use the Test Impact analysis or Pareto testing, you'll need to provide test coverage on a per-test basis to Teamscale with a testwise coverage report.

Depending on the technologies you use, select the right approach to record testwise coverage:

• Generic Approach with teamscale-build
• Java Virtual Machine (JVM)
  • Gradle
  • Maven and JUnit 5

Generic Approach with teamscale-build

This approach works for all technologies.

1. You instrument your tests with a supported coverage profiler.
2. You run each of your tests separately, gather test coverage for only that one test. You end up with many test coverage reports, one per test case.
3. You use teamscale-build to combine these report files into one testwise coverage report.
4. You upload the testwise coverage report with teamscale-upload.

Java Virtual Machine (JVM)

Our Java profiler can record testwise coverage. For tests that are run by Maven or Gradle, we provide plugins that automate both testwise coverage collection and smart test execution with Test Impact analysis.

For other tests, you can use our profiler directly and send events via its REST API when tests start and end.

Gradle

For Gradle based builds with JUnit Platform based tests there also exists a Gradle plugin. The plugin configures the test execution to use a custom Test Engine which handles the communication with Teamscale and delegates the actual execution to existing Test Engines.

Tutorial

We also have a Tutorial on how to set up Test Impact Analysis in a Gradle-based Java project.

Maven and JUnit 5

If you use JUnit 5 (or any test framework built on JUnit 5, e.g., Serenity) for your tests, you can use our teamscale-maven-plugin to easily run TIA in your builds. It configures Surefire and Failsafe, so they fetch the list of impacted tests from Teamscale, execute them, and record and upload Testwise Coverage. This also works for Spring Boot applications.

We recommend that for regular, fast feedback, you run only impacted tests for most of your builds. It is, however, necessary to also run all tests from time to time (e.g., in a nightly or weekly build). This allows Teamscale to regularly update Testwise Coverage information for all tests. To achieve this, you can use Maven profiles and enable the teamscale-maven-plugin only in a special profile.

<profiles>
	<profile>

		<id>tia</id>
		<activation>
			<property>
				<name>tia</name>
			</property>
		</activation>

		<dependencies>
			<dependency>
				<groupId>com.teamscale</groupId>
				<artifactId>impacted-test-engine</artifactId>
				<version>30.0.0</version>
				<scope>test</scope>
			</dependency>
		</dependencies>

		<build>
			<plugins>
				<plugin>
					<groupId>com.teamscale</groupId>
					<artifactId>teamscale-maven-plugin</artifactId>
					<version>30.0.0</version>
					<executions>
						<execution>
							<goals>
								<goal>prepare-tia-unit-test</goal> <!-- Enables TIA for Surefire unit tests -->
								<goal>prepare-tia-integration-test
								</goal> <!-- Enables TIA for Failsafe integration tests -->
							</goals>
						</execution>
					</executions>
					<configuration>
						<teamscaleUrl>http://localhost:8080</teamscaleUrl>
						<projectId>teamscale-project-id</projectId>
						<username>build</username>
						<accessToken>${env.TEAMSCALE_ACCESS_TOKEN}</accessToken>
						<includes>
							<include>*com.your.company.*</include>
						</includes>
					</configuration>
				</plugin>
			</plugins>
		</build>
	</profile>
</profiles>

List of All Available Parameters

To list all available parameters run mvn com.teamscale:teamscale-maven-plugin:help -Ddetail=true (also works without a maven project).

This adds the plugin and configures the connection to Teamscale. We recommend you also set at least one include pattern so coverage is only recorded for your code, not for 3rd party libraries.

It also configures Surefire and Failsafe to use our impacted-test-engine. This engine automatically fetches impacted tests from Teamscale and executes them. Make sure the engine version matches the teamscale-maven-plugin version.

Parallel Tests Not Supported

Parallelized tests are not supported by this plugin. When tests run in parallel, it is impossible to record Testwise Coverage as the coverage from the tests running in parallel cannot be distinguished anymore. Make sure to set <forkCount>1</forkCount> and <threadCount>1</threadCount> (c.f. the Maven documentation) for all tests where you enable the provider, otherwise a TIA-enabled build will fail. These settings are the defaults for Surefire and Failsafe.

Use of <reuseForks>true</reuseForks>

It is strongly discouraged to set<reuseForks>true</reuseForks> as this will start and close a new JVM for each test case. Since the Teamscale-Maven-Plugin writes a coverage report every time the JVM closes, this can lead to performance problems on some systems.

The plugin sets the right system property to attach itself to Surefire and Failsafe tests. Surefire and Failsafe pick up the system property automatically and apply it to any JVMs they spawn.

Spring Boot

The plugin also works out-of-the-box for Spring Boot applications. Here, the Spring Boot Maven plugin will pick up the JVM options and pass them to the Spring Boot application that is started before the integration tests.

In some cases, your system-under-test is not launched by these plugins but by some other means (e.g., another plugin). In this case, make sure the argLine property is passed to the JVM of your system-under-test. If you do not want Surefire or Failsafe to pick up the argLine automatically or want to add additional parameters, you can change the name of the property that holds the JVM arguments by setting the configuration property <propertyName>.

If you need to add additional parameters to the Teamscale JaCoCo agent, you can use <additionalAgentOptions>.

Troubleshooting

The plugin creates a log file in target/tia/agent.log. You can enable debug logging by adding the <debugLogging>true</debugLogging> configuration property.

If the log file is not created when executing your tests, check if you have <argLine>...</argLine> in your pom.xml and remove it. This overwrites the arguments we set in the plugin and thus de-registers the coverage profiler.

In some cases, the test reports generated by Surefire/Failsafe may also contain error information. You can find them under target/surefire-reports after all tests have finished. Check the stdout/stderr of failing tests for additional error messages.

If the build fails after applying the above POM changes but works without them, try to re-run the build with -e to discover the error that caused this.

If you notice largely increased test runtimes on systems that set <reuseForks>true</reuseForks> it is recommended to set the <tiaMode>exec-file</tiaMode>. Writing a testwise coverage report every time the JVM closes can be time-consuming, so writing raw .exec files instead, has been shown to increase test execution times. There is a separate Maven goal com.teamscale:teamscale-maven-plugin:testwise-coverage-converter that can then be run to batch convert the created raw .exec files into a testwise coverage report. You can use the com.teamscale:teamscale-maven-plugin:upload-coverage goal to then upload the created testwise coverage report to Teamscale.