Teamscale Java Profiler - Testwise Coverage Recording

The Teamscale Java Profiler can be used to record testwise coverage which is needed to set up Test Impact Analysis or Pareto with Teamscale. If you are unfamiliar with Test Impact Analysis, please have a look at our TIA tutorial and how-to first.

Gradle & Maven

If you want to set up Test Impact Analysis and are using Gradle or Maven, it is usually easier to use our respective Gradle or Maven plugin.

Setup

To enable testwise coverage collection, the mode option must be set to testwise. Furthermore, the test system (the application executing the test specification) has to inform the profiler of when a test started and finished via the REST API. To be able to send the profiler test events, you must set the following additional options in the profiler configuration:

• http-server-port: the port at which the profiler should start an HTTP server that listens for test events (Recommended port is 8123)
• class-dir: the path under which all class files of the profiled application are stored on the test environment. Can be a directory or a Jar/War/Ear/... file. Separate multiple paths with a semicolon (for details, please see the path format section).

To record testwise coverage, you need to call the profiler API endpoints in the following order:

1. (Optionally: [POST] /testrun/start to get a list of impacted tests)
2. [POST] /test/start
3. [POST] /test/end
4. Repeat step 2 and 3 until all the tests ran through
5. [POST] /testrun/end

Have a look at the API section for more explanations about these endpoints. For additional configuration options, have a look at the options section.

API Endpoints

These are the relevant API endpoints to generate testwise coverage reports. Have a look at our advanced setup page for more information on the profilers REST API.

• [GET] /test Returns the testPath of the current test. The result will be empty when the test already finished or was not started yet.

• [POST] /testrun/start If you configured a connection to Teamscale via the teamscale- options, this will fetch impacted tests from Teamscale and return them in the response body. You may optionally provide a list of all available test cases in the body of the request. These will also be used to generate the testwise coverage report in [POST] /testrun/end. The format of the request body is:

    [
        {
          "clusterId": "<ID of the cluster the test belongs to>",
          "uniformPath": "<Unique name of the test case>",
          "sourcePath": "<Optional: Path to the source of the test>",
          "content": "<Optional: Value to detect changes to the test, e.g. hash code, revision, ...>"
        }
    ]

  Additionally, you may pass the following optional URL query parameters:

  • include-non-impacted: If this is true, will not perform test-selection, only test-prioritization.
  • baseline: The baseline in the form branch:timestamp to indicate the time since which changes should be considered. If not given, the time since the last uploaded testwise coverage report is used.

• [POST] /testrun/end?partial=true If you configured a connection to Teamscale via the teamscale- options and enabled tia-mode=teamscale-upload, this will upload a testwise coverage report to Teamscale. partial describes whether the recorded tests represent a subset of all tests i.e. only impacted tests were executed. This tells Teamscale to keep tests even if they are not present in the latest report. Defaults to false.

• [POST] /test/start/{uniformPath} Signals to the profiler that the test with the given uniformPath is about to start.

• [POST] /test/end/{uniformPath} Signals to the profiler that the test with the given uniformPath has just finished. The body of the request may optionally contain the test execution result in json format:

  {
    "result": "ERROR",
    "message": "<stacktrace>|<ignore reason>"
  }

  result can be one of:

  • PASSED Test execution was successful.
  • IGNORED The test is currently marked as "do not execute" (e.g. JUnit @Ignore or @Disabled).
  • SKIPPED Caused by a failing assumption.
  • FAILURE Caused by a failing assertion.
  • ERROR Caused by an error during test execution (e.g. exception thrown).

UniformPath

The uniformPath parameter must be url encoded. E.g. com/example/MyTest/testSomething -> http://localhost:8123/test/start/com%2Fexample%2FMyTest%2FtestSomething.

Options

You can select how the testwise coverage report is generated by setting the tia-mode option to either:

• teamscale-upload: the profiler will upload the testwise report to Teamscale once you call the POST /testrun/end REST endpoint. This option is the most convenient of the different modes as the profiler handles all aspects of report generation and the upload to Teamscale for you. This mode may slow down the startup of the system under test and result in a larger memory footprint than the exec-file mode.
• exec-file (default): The profiler stores the coverage in a binary *.exec file within the out directory. This is most useful when running tests in a CI/CD pipeline where the build tooling can later batch-convert all *.exec files and upload a testwise coverage report to Teamscale or in situations where the profiler must consume as little memory and CPU as possible and thus cannot convert the execution data to a report as required by the other options. It is, however, less convenient as you have to convert the *.exec files yourself.
• disk: The profiler stores the testwise coverage in JSON *.json files within the out directory. This is most useful when running tests in a CI/CD pipeline where the testwise coverage is to be provided in storage systems (such as S3) using custom upload scripts.
• http: the profiler converts the coverage collected during a test in-process and returns it as a JSON in the response to the [POST] /test/end/... request. This allows the caller to handle merging coverage of multiple tests into one testwise coverage report, e.g. in situations where more than one profiler is running at the same time (e.g. profiling across multiple microservices.) This option may slow down the startup of the system under test and result in a larger memory footprint than the exec-file mode. The response will be part of a testwise coverage report, without the surrounding tests array.

To convert the .exec files produced by tia-mode=exec-file, you can run the bin/convert tool. Run bin/convert --help to see all available command line options.