# Test Impact Analysis for Java

This tutorial will show you how to set up Test Impact Analysis (TIA) for the most simple Java project with Gradle and JUnit 5. It will teach you how to use the Teamscale Gradle plugin and configure it for the TIA so it uploads testwise coverage and controls which tests are run in your CI/CD pipeline. We will then run through several common code changes and learn how the TIA handles them.

# TIA Setup

This tutorial is based on this very simple Java project hosted on GitHub. During the tutorial, you'll need to make changes to this repository and push them to GitHub.Thus, please fork it to your own GitHub account and clone it to your local machine:

git clone https://github.com/<path-to-your-repo>

We have already done every step of this tutorial for you. The result of every step of this tutorial has a corresponding Git branch in the repository. Each branch builds on the branch of the previous step. So if you'd like to see what e.g. step 5 should look like, simply run git checkout step5.

What to do when forking is not possible

If you cannot fork the project, you can use our project but keep in mind, that you cannot push to GitHub then. In order for TIA to work correctly in this scenario, you can't do the changes yourself but you have to checkout the branches for the individual steps.

To see all the changes made to the project between two steps, simply run a git diff:

git diff origin/step1 origin/step2

Initially, master is checked out, which contains our unmodified simple Java project. The project contains one class Calculator and its corresponding JUnit 5 test class CalculatorTest.

You can run these existing tests with ./gradlew clean check. Notice that it runs both test cases.

# Step 0: Importing the Project into Teamscale

Before we can do anything in the Gradle project, we first need to import it into Teamscale. If you haven't done so already, please download, install and start Teamscale.

Next, please download the following Teamscale backup:

Teamscale Backup for This Tutorial

It simply sets the admin user's access key so you don't need to adjust that in the Gradle script at every step of the tutorial. To this end, import the backup into Teamscale.

Finally, create a Teamscale project for the Git repository that you forked above.

  • Name the project tia-junit5 (this value is later used in the Gradle scripts).
  • Use Teamscale's default Java analysis profile.
  • Add a Git connector and in its settings add an account and enter the URL https://github.com/<path-to-your-repo. You can freely choose any name for the account and you can leave username and password empty.
  • Enable the option Enable branch analysis so Teamscale analyzes all branches of the repository.
  • Set the polling interval to 1 (second) to make sure Teamscale always knows about the latest code changes for this tutorial.
  • Leave all other options at their default values.

Save the project. You can then go to the Metrics perspective in Teamscale and examine the imported source code.

# Step 1: Adding the Teamscale Plugin

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/master origin/step1`.

The teamscale-gradle-plugin will record testwise coverage and allows you to easily configure TIA for your project. We will now add it to our Java project.

First, add a build script dependency on the teamscale-gradle-plugin at the start of the file build.gradle:




 


 


buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'com.teamscale:teamscale-gradle-plugin:17.0.0'
    }
}

Next, we apply and configure the plugin so it knows where our Teamscale instance is running and how to access it. Add the following to the body of build.gradle:

apply plugin: 'teamscale'

teamscale {

    server {
        url = 'http://localhost:8080/'
        userName = 'admin'
        userAccessToken = 'q4tu9vfAAjQZ1peCpPvQHSrLi5CeIcGY'
        project = 'tia-junit5'
    }

    report {
        partition = 'Unit Tests'
        testwiseCoverage()
    }

}

This configuration will be used by all Gradle tasks that run the TIA. If you're wondering what partition is for, please check our glossary entry for partitions.

# Step 2: Collecting Testwise Coverage for All Tests

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step1 origin/step2

To continue with the tutorial from this point, run

git checkout step1
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

Before we can run the TIA, we first need to once collect testwise coverage for all our tests. This ensures that Teamscale knows about every existing test and its coverage.

To be able to do this, we create a new Gradle task that runs the TIA for us. Add the following to the body of build.gradle:

tasks.register('tiaTests', com.teamscale.TestImpacted) {
    jacoco {
        includes = ["tia.*"]
    }
}

Running a Subset of Your Tests with TIA

The com.teamscale.TestImpacted task type has the same options as a normal Gradle Test task. So you can use these to select, for which tests Teamscale should perform testwise coverage recording and test selection via the TIA. In most real setups, you won't want to use TIA for all your tests. E.g. unit tests are usually fast enough that you don't need to use TIA on them. But maybe your automated UI tests benefit from the reduced run-time that TIA provides instead.

This task functions exactly like a normal test task. It will execute all our tests by default. We also set an include pattern so only our code is profiled, not e.g. the libraries that we use or our test framework.

Always Set Include Patterns

By setting include patterns, you ensure that only your code is profiled. This makes your tests run faster and the resulting coverage report file smaller. Always set include patterns so only your packages are being profiled.

Try out the new task now by running

./gradlew clean tiaTests

However, we can also ask it to record testwise coverage for us by passing the --impacted --run-all-tests command line switches:

./gradlew clean tiaTests --impacted --run-all-tests

A testwise coverage report has now been generated for both of our test cases in build/reports/testwise_coverage/testwise_coverage-Unit-Tests-tiaTests.json. Open it in your favourite editor and see that it contains our two test cases, their respective test coverage, whether the tests failed or passed and how long it took to run them.

Next, we'll upload this information to Teamscale. The teamscale-gradle-plugin already contains a task to do this:

./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

teamscaleReportUpload cannot be run standalone

You can only run this task if you have previously run a TestImpacted task. By itself, teamscaleReportUpload doesn't do anything.

You can then go to Teamscale and see the results for yourself: Change to the Metrics perspective, then select it's Test Results view in the sidebar. At the top, in the branch chooser, select the step2 branch to which the testwise coverage was uploaded. There you'll see that Teamscale now knows about our latest test run:

Our uploaded testwise coverage and test run metadata

Teamscale is Branch-Aware

Teamscale knows about all the branches in your VCS and will automatically handle test executions for any number of them correctly. So you can e.g. run tests on a feature branch and when you merge it to master, Teamscale will automatically take care to also merge your testwise coverage and test results.

If you click on the CalculatorTest, you'll see a list of all test cases of that class. After clicking on one of them, you can see further details like a treemap with that test case's testwise coverage and the test run metadata that the testwise coverage report contained:

Details for one test case

# Common TIA Scenarios

In this part, we'll go through several common scenarios to show-case how the TIA intelligently selects tests in each of them to minimize test runtime while also maximizing the amount of bugs found by your tests.

# Step 3: If We Change One Method, Only the Corresponding Tests Are Run

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step2 origin/step3

To continue with the tutorial from this point, run

git checkout step2
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

Now let's see TIA in action! We'll change the method Calculator.sumImpl() which is called by Calculator.sum(). This change only affects one of our two test cases: CalculatorTest.testSum() so we'd expect the TIA to only run that test and not CalculatorTest.testMinus().

Let's introduce a not-so-subtle bug in the method:






 



    public int sum(int a, int b) {
        return sumImpl(a, b);
    }

    private int sumImpl(int a, int b) {
        a = a * 5;
        return a + b;
    }

Commit and push the changes.

Now let's run TIA. For this, we use our TestImpacted task and this time pass the --impacted --continue command line switches. --impacted instructs the teamscale-gradle-plugin to only run the test cases that Teamscale reports as impacted by our changes. --continue instructs Gradle to upload the testwise coverage even when a test fails.

./gradlew clean tiaTests --impacted --continue teamscaleReportUpload

Always Upload Testwise Coverage

We recommend you always run the teamscaleReportUpload task after executing impacted tests. This keeps Teamscale's TIA data up-to-date. The data is recorded either way, so there is no reason not to make use of it!

Gradle's test summary on the console shows, as we expected, that only one test was executed while the other was ignored in this test run. And it found the bug we introduced.






 

> Task :tiaTests FAILED

CalculatorTest > testSum() FAILED
    org.opentest4j.AssertionFailedError at CalculatorTest.java:9

2 tests completed, 1 failed, 1 skipped

# Step 4: If a Test Case Fails Once, It Is Rerun Until It Succeeds

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step3 origin/step4

To continue with the tutorial from this point, run

git checkout step3
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

In the last step, we made some changes to our code which made a test case fail. The TIA picked this up and ran the test case as expected. However, what happens when we now do some unrelated changes that don't fix the test failure? What we don't want to happen is that TIA now ignores the failing test case. After all, it still needs attention. Let's try this out: Let's change the message of the println in the minus() method:


 



    public int minus(int a, int b) {
        System.out.println("tia rocks");
        return a - b;
    }

Commit and push the changes.

Normally, TIA would only run the testMinus() test case as the testSum() test case is not affected by this change. However, since testSum() failed the last time we ran it, it is re-run this time again:











 

./gradlew clean tiaTests --impacted --continue teamscaleReportUpload


> Task :tiaTests FAILED

tia.CalculatorTest > testSum() FAILED
    org.opentest4j.AssertionFailedError at CalculatorTest.java:11

tia.CalculatorTest > testMinus() PASSED

2 tests completed, 1 failed

# Step 5: If We Add a New Test Case, It Is Always Run

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step4 origin/step5

To continue with the tutorial from this point, run

git checkout step4
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

Let's undo the change that causes the test failure and also add a new test case for the sum() method. This will cause the testSum() test to be re-run. At the same time, let's add a new test-case for the minus() method:

    @Test
    public void testMinus2() {
        assertEquals(-1, new Calculator().minus(1, 2));
    }

Commit and push the changes.

How does TIA deal with newly added test cases? After all: it doesn't know yet what coverage this test will produce. So we'd expect it to always run any new test cases, just in case they might fail:










./gradlew clean tiaTests --impacted --continue teamscaleReportUpload


> Task :tiaTests

tia.CalculatorTest > testMinus2() PASSED

tia.CalculatorTest > testSum() PASSED

As you can see, both the test case for the modified sum() method is run and our new testMinus2(). All this is to ensure that we don't miss any bugs that our newly modified test cases would have caught.

# Step 6: What about Refactorings?

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step5 origin/step6

To continue with the tutorial from this point, run

git checkout step5
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

If we just use the IDE to refactor some code, there's really no need to re-run its test cases. Most importantly: if we rename something, TIA should detect that this is not an actual test-worthy change but just a rename refactoring. Let's try it by renaming some method parameters:

 



    public int sum(int first, int second) {
        return sumImpl(first, second);
    }

Commit and push the changes.

Now run TIA again:

./gradlew clean tiaTests --impacted --continue teamscaleReportUpload


BUILD SUCCESSFUL in 2s
6 actionable tasks: 6 executed

As you can see, no tests were executed as nothing test-worthy changed in the system. The same is true for non-code changes like adding or removing whitespace or changing code comments.

# Step 7: What Happens if Teamscale is Offline?

If You Want to Skip This Step

To see the changes performed in this step, run

git diff origin/step6 origin/step7

To continue with the tutorial from this point, run

git checkout step6
./gradlew clean tiaTests --impacted --run-all-tests teamscaleReportUpload

One important failure case to think about is: what if your build server can't reach your Teamscale instance? This might e.g. happen when your network isn't working, someone changed the firewall rules by accident or someone shut down Teamscale for some maintenance work. In neither case do you want your CI/CD pipeline to crash and stop working! So let's simulate this:

Simply kill the Teamscale process in your task manager.

In this step, we change the sumImpl() method and introduce another bug:


 


    private int sumImpl(int a, int b) {
        return a - b;
    }

Commit and push the changes.

If TIA were running, it would re-run only the tests for sum(). Let's see what happens now that Teamscale is offline:

./gradlew clean tiaTests --impacted --continue teamscaleReportUpload


> Task :tiaTests FAILED

tia.CalculatorTest > testSum() FAILED
    org.opentest4j.AssertionFailedError at CalculatorTest.java:11

tia.CalculatorTest > testMinus() PASSED

tia.CalculatorTest > testMinus2() PASSED

3 tests completed, 1 failed

As you can see, the teamscale-gradle-plugin simply ran all tests. This is the default behaviour, in case anything goes wrong, including when Teamscale is not reachable. In that case, your CI/CD pipeline will just safely revert to its old behaviour so you don't miss any bugs.