Test Impact Analysis for Java
This tutorial will show you how to set up Test Impact Analysis (TIA) for the most simple Java project with Gradle and JUnit 5. It will teach you how to use the Teamscale Gradle plugin and configure it for the TIA so it uploads Testwise Coverage and controls which tests are run in your CI/CD pipeline. We will then run through several common code changes and learn how the TIA handles them.
TIA Setup
This tutorial is based on this very simple Java project hosted on GitHub. During the tutorial, you'll need to make changes to this repository and push them to GitHub. Thus, please fork it to your own GitHub account and clone it to your local machine:
git clone https://github.com/<path-to-your-repo>We have already done every step of this tutorial for you. The result of every step of this tutorial has a corresponding Git branch in the repository. Each branch builds on the branch of the previous step. So if you'd like to see what, for example, step 5 should look like, simply run git checkout step5.
What to do when forking is not possible
If you cannot fork the project, you can use our project but keep in mind, that you cannot push to GitHub then. In order for TIA to work correctly in this scenario, you can't do the changes yourself but you have to checkout the branches for the individual steps. When inspecting the coverage in Teamscale, you will also have to select the corresponding branch.
To see all the changes made to the project between two steps, simply run a git diff:
git diff origin/step1 origin/step2Initially, master is checked out, which contains our unmodified Java project. The project contains one class Calculator and its corresponding JUnit 5 test class CalculatorTest.
You can run these existing tests with ./gradlew clean check. Notice that it runs both test cases.
Step 0: Importing the Project into Teamscale
Before we can do anything in the Gradle project, we first need to import it into Teamscale. If you haven't done so already, please download, install and start Teamscale.
Next, please download the following Teamscale backup:
It simply sets the admin user's access key so you don't need to adjust that in the Gradle script at every step of the tutorial. To this end, import the backup into Teamscale.
Finally, create a Teamscale project for the Git repository that you forked above.
- Name the project
tia-junit5(this value is later used in the Gradle scripts). - Use Teamscale's default Java analysis profile.
- Set the Branching Configuration to
.*>2020-01-01 - Add a Git connector and in its settings add an account and enter the URL
https://github.com/<path-to-your-repo>. You can freely choose any name for the account, and you can leave username and password empty. - Enable the option
Enable branch analysisso Teamscale analyzes all branches of the repository. - Set the polling interval to 1 (second) to make sure Teamscale always knows about the latest code changes for this tutorial.
- Leave all other options at their default values.
Save the project. You can then go to the Metrics perspective in Teamscale and examine the imported source code.
Step 1: Adding the Teamscale Plugin
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/master origin/step1The teamscale-gradle-plugin will record Testwise Coverage and allows you to easily configure TIA for your project. We will now add it to our Java project.
We apply and configure the plugin, so it knows where our Teamscale instance is running and how to access it. For a reference of all configuration options, see TeamscalePluginExtension. Add the following to the body of build.gradle(.kts):
plugins {
id "com.teamscale" version "34.0.1"
}
teamscale {
server {
url = "http://localhost:8080/"
userName = "admin"
userAccessToken = "q4tu9vfAAjQZ1peCpPvQHSrLi5CeIcGY"
project = "tia-junit5"
}
report {
testwiseCoverage {
partition = "Unit Tests"
}
}
}This configuration will be used by all Gradle tasks that run the TIA. If you're wondering what partition is for, please check our glossary entry for partitions.
In a multi-module Gradle setup, you would usually apply the plugin to each module by adding it to your convention plugin.
Step 2: Collecting Testwise Coverage for All Tests
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step1 origin/step2Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step1Before we can run the TIA, we first need to once collect Testwise Coverage for all our tests. This ensures that Teamscale knows about every existing test and its coverage.
To be able to do this, we create a new Gradle task that runs the TIA for us. Add the following to the body of build.gradle:
tasks.register("tiaTests", com.teamscale.TestImpacted) {
useJUnitPlatform()
testClassesDirs = testing.suites.test.sources.output.classesDirs
classpath = testing.suites.test.sources.runtimeClasspath
jacoco {
includes = ["tia.*"]
}
}Running a Subset of Your Tests with TIA
The com.teamscale.TestImpacted task type has the same options as a normal Gradle Test task. So you can use these to select, for which tests Teamscale should perform Testwise Coverage recording and test selection via the TIA. In most real setups, you won't want to use TIA for all your tests. E.g. unit tests are usually fast enough that you don't need to use TIA on them. But maybe your automated UI tests benefit from the reduced run-time that TIA provides instead.
This task functions exactly like a normal test task. It will execute all our tests by default. We also set an include pattern so only our code is profiled and not the libraries that we or our test framework use.
Always Set Include Patterns
By setting include patterns, you ensure that only your code is profiled. This makes your tests run faster and the resulting coverage report file smaller. Always set include patterns so only your packages are being profiled.
Try out the new task by running
./gradlew clean tiaTestsA Testwise Coverage report has now been generated for both of our test cases in build/reports/testwise-coverage/tiaTests/Unit-Tests.json. Open it in your favourite editor and see that it contains our two test cases, their respective test coverage, whether the tests failed or passed and how long it took to run them.
Next, we'll upload this information to Teamscale. The teamscale-gradle-plugin already contains a task teamscaleReportUpload to do this:
./gradlew clean tiaTests teamscaleReportUploadteamscaleReportUpload cannot be run standalone
You can only run this task if you have previously run a TestImpacted task. teamscaleReportUpload uploads previously generated reports (coverage and test results) to Teamscale. If no reports were generated, teamscaleReportUpload does not do anything.
You can then go to Teamscale and see the results for yourself: Change to the Metrics perspective, then select it's Tests view in the sidebar. There you'll see that Teamscale now knows about our latest test run:
Teamscale is Branch-Aware
Teamscale knows about all the branches in your VCS and will automatically handle test executions for any number of them correctly. So you can e.g. run tests on a feature branch and when you merge it to master, Teamscale will automatically take care to also merge your Testwise Coverage and test results.
If you click on the CalculatorTest, you'll see all test cases of that class. After clicking on one of them, you can see further details like a treemap with that test case's Testwise Coverage and the test run metadata that the Testwise Coverage report contained:
Common TIA Scenarios
In this part, we'll go through several common scenarios to showcase how the TIA intelligently selects tests in each of them to minimize test runtime while also maximizing the amount of bugs found by your tests.
Step 3: If We Change One Method, Only the Corresponding Tests Are Run
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step2 origin/step3Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step2
./gradlew clean tiaTests --continue teamscaleReportUploadNow let's see TIA in action! We'll change the method Calculator.sumImpl() which is called by Calculator.sum(). This change only affects one of our two test cases: CalculatorTest.testSum() so we'd expect the TIA to only run that test and not CalculatorTest.testMinus().
Let's introduce a not-so-subtle bug in the method:
public int sum(int a, int b) {
return sumImpl(a, b);
}
private int sumImpl(int a, int b) {
a = a * 5;
return a + b;
}Commit and push the changes.
Now let's run TIA. For this, we use our TestImpacted task and this time pass the --impacted --continue command line switches. --impacted instructs the teamscale-gradle-plugin to only run the test cases that Teamscale reports as impacted by our changes. --continue instructs Gradle to upload the Testwise Coverage even when a test fails.
./gradlew clean tiaTests --impacted --continue teamscaleReportUploadAlways Upload Testwise Coverage
We recommend to always run the teamscaleReportUpload task after executing impacted tests. This keeps Teamscale's TIA data up-to-date. The data is recorded either way, so there is no reason not to make use of it!
Gradle's test summary on the console shows, as we expected, that only one test was executed while the other was ignored in this test run. And it found the bug we introduced.
> Task :tiaTests FAILED
JUnit Jupiter > CalculatorTest > testSum() FAILED
org.opentest4j.AssertionFailedError at CalculatorTest.java:11
1 test completed, 1 failedStep 4: If a Test Case Fails Once, It Is Rerun Until It Succeeds
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step3 origin/step4Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step3
./gradlew clean tiaTests --continue teamscaleReportUploadIn the last step, we made some changes to our code which made a test case fail. The TIA picked this up and ran the test case as expected. However, what happens when we now do some unrelated changes that don't fix the test failure? What we don't want to happen is that TIA now ignores the failing test case. After all, it still needs attention. Let's try this out: Let's change the message of the println in the minus() method:
public int minus(int a, int b) {
System.out.println("gi");
System.out.println("tia rocks");
return a - b;
}Commit and push the changes.
Normally, TIA would only run the testMinus() test case as the testSum() test case is not affected by this change. However, since testSum() failed the last time we ran it, it is re-run this time again:
./gradlew clean tiaTests --impacted --continue teamscaleReportUpload
> Task :tiaTests FAILED
JUnit Jupiter > CalculatorTest > testSum() FAILED
org.opentest4j.AssertionFailedError at CalculatorTest.java:11
JUnit Jupiter > CalculatorTest > testMinus() PASSED
2 tests completed, 1 failedStep 5: If We Add a New Test Case, It Is Always Run
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step4 origin/step5Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step4
./gradlew clean tiaTests --continue teamscaleReportUploadLet's undo the change that causes the test failure and also add a new test case for the sum() method. This will cause the testSum() test to be re-run. At the same time, let's add a new test-case for the minus() method:
@Test
public void testMinus2() {
assertEquals(-1, new Calculator().minus(1, 2));
}Commit and push the changes.
How does TIA deal with newly added test cases? After all: it doesn't know yet what coverage this test will produce. So we'd expect it to always run any new test cases, just in case they might fail:
./gradlew clean tiaTests --impacted --continue teamscaleReportUpload
> Task :tiaTests
JUnit Jupiter > CalculatorTest > testSum() PASSED
JUnit Jupiter > CalculatorTest > testMinus2() PASSEDAs you can see, both the test case for the modified sum() method is run and our new testMinus2(). All this is to ensure that we don't miss any bugs that our newly modified test cases would have caught.
Step 6: What about Refactorings?
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step5 origin/step6Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step5
./gradlew clean tiaTests --continue teamscaleReportUploadIf we just use the IDE to refactor some code, there's really no need to re-run its test cases. Most importantly: if we rename something, TIA should detect that this is not an actual test-worthy change but just a rename refactoring. Let's try it by renaming some method parameters:
public int sum(int a, int b) {
public int sum(int first, int second) {
return sumImpl(a, b);
return sumImpl(first, second);
}Commit and push the changes.
Now run TIA again:
./gradlew clean tiaTests --impacted --continue teamscaleReportUpload
BUILD SUCCESSFUL in 2s
6 actionable tasks: 6 executedAs you can see, no tests were executed as nothing test-worthy changed in the system. The same is true for non-code changes like adding or removing whitespace or changing code comments.
Step 7: What Happens if Teamscale is Offline?
Inspect Changes Made in This Step
To see the changes performed in this step, run
git diff origin/step6 origin/step7Skip Previous Steps
To continue with the tutorial from this point, run
git checkout master
git merge origin/step6
./gradlew clean tiaTests --continue teamscaleReportUploadOne important failure case to think about is: what if your build server can't reach your Teamscale instance? This might e.g. happen when your network isn't working, someone changed the firewall rules by accident or someone shut down Teamscale for some maintenance work. In neither case do you want your CI/CD pipeline to crash and stop working! So let's simulate this:
Simply kill the Teamscale process in your task manager.
In this step, we change the sumImpl() method and introduce another bug:
private int sumImpl(int a, int b) {
return a + b;
return a - b;
}Commit and push the changes.
If TIA were running, it would re-run only the tests for sum(). Let's see what happens now that Teamscale is offline:
./gradlew clean tiaTests --impacted --continue teamscaleReportUpload
> Task :tiaTests FAILED
JUnit Jupiter > CalculatorTest > testSum() FAILED
org.opentest4j.AssertionFailedError at CalculatorTest.java:11
JUnit Jupiter > CalculatorTest > testMinus() PASSED
JUnit Jupiter > CalculatorTest > testMinus2() PASSED
3 tests completed, 1 failedAs you can see, the teamscale-gradle-plugin simply ran all tests. This is the default behaviour, in case anything goes wrong, including when Teamscale is not reachable. In that case, your CI/CD pipeline will just safely revert to its old behaviour so you don't miss any bugs.