# Testwise Coverage

# Report Format

Testwise Coverage is a Teamscale-specific coverage format that allows to express coverage information on a per test level.

Versions

There are two versions of this report format. Version 2 is supported in Teamscale 7.8 and later. Version 1 is still supported for backwards compatibility. This document describes the version 2 of the Testwise Coverage report format. You can still find the documentation of the version 1 format here.

# Preliminaries

  • Uniform path
    • File path like string that represents a hierarchy. Parts of the hierarchy are delimited by slashes.
  • Line range string
    • String containing line ranges and individual lines separated by commas.
    • Compact representation of a set of lines of code.
    • Example: "10-12,13,22-27,40"

# TestwiseCoverageReport

The top level JSON object is the TestwiseCoverageReport. It contains the following attributes:

versionintrequired
  • The version for the format described here should be 2.
  • Version number for the Testwise Coverage report version.
  • If not given the parser will fall back to version 1 of the Testwise Coverage format.
partialbooloptional
  • If set to true the set of tests contained in the report don't represent the full set of tests. These tests are added or updated in Teamscale, but no tests that are missing in the report will be deleted.
  • If not provided defaults to false.
filesArray<FileInfo>required
  • See section for FileInfo.
  • FileInfos are referenced elsewhere in the report by their index within the array.
executionUnitsArray<ExecutionUnit>optional
  • See section for ExecutionUnit.
  • Defaults to the empty array if omitted.
testsArray<TestInfo>optional
  • See section for TestInfo.
  • Defaults to the empty array if omitted.

# FileInfo

Information about a covered file. It contains the following attributes.

pathstringrequired
  • Code path within the repository.
  • Ideally the exact relative path from the repository root. Otherwise, the path will be resolved to the longest matching suffix in Teamscale.
coverableLinesstringoptional
  • Line range string for the lines in the source code that are considered coverable by the coverage tool, i.e. are reported with execution counts =0 or >0.
  • If omitted Teamscale will determine coverable lines heuristically based on source code.

# TestInfo

uniformPathstringrequired
  • A unique hierarchical identifier of the test with / as separator.
  • If this is a parameterized test the test may occur multiple times with different parameters.
  • E.g. Java package name / test class name / test method name
hashstringoptional
  • A hash determining the version of the test implementation.
  • E.g. SHA-1 hash of a test description XML or a revision number of the test specification.
  • If the value changes the test is considered as impacted.
durationSecondsfloatrequired The duration in seconds it took to execute the test.
resultstringrequired Describes the result of the test execution. Can be one of:
  • PASSED Test execution was successful.
  • IGNORED The test is currently marked as "do not execute" (e.g. JUnit @Ignore or @Disabled).
  • SKIPPED Caused by a failing assumption.
  • FAILURE Caused by a failing assertion.
  • ERROR Caused by an error during test execution (e.g., exception thrown).
Tests that did not pass previously are always selected for test execution.
messagestringoptional Error message and stacktrace or skip reason.
coverageMap<string,string>optional A map of strings to line range strings. The keys are the integer indexes in string form of the FileInfo objects in the top-level files array. The values are the line range strings representing the covered lines.

# ExecutionUnit

Represents a set of tests that is executed together. An execution unit in the Testwise Coverage report can either contain a subset or all of the tests that are executed as one unit. Examples include test binaries built for C/C++ code or test Jar files for Java code. During the build process, all tests included in an execution unit will be executed at once to avoid overhead. It contains the following attributes.

uniformPathstringrequired
  • A unique hierarchical identifier of the test with / as separator.
  • E.g. unique path to executable or Jar file.
hashstringoptional
  • A hash determining the version of the execution unit.
  • E.g. SHA-1 hash of test resources.
  • If the value changes the execution unit is considered to be impacted.
durationSecondsfloatrequired The duration in seconds it took to execute the execution unit.
resultstringrequired
  • The overall result of the execution of the execution unit. Can be one of:
    • PASSED Execution was successful.
    • IGNORED The execution unit is currently ignored.
    • SKIPPED Caused by a failing assumption.
    • INCONCLUSIVE Test assertions cannot be verified.
    • FAILURE Caused by a failing assertion.
    • ERROR Caused by an error during test execution (e.g., exception thrown).
  • May differ from the combined result of tests contained in the execution unit.
  • If the value changes the execution unit is considered to be impacted.
Execution units that did not pass previously are always selected for execution in the context of Test Impact Analysis.
coverageMap<string,string>required
  • A map of strings to line range strings. The keys are the integer indexes in string form of the FileInfo objects in the top-level files array. The values are the line range strings representing the covered lines.
  • The coverage produced by executing all the tests in the execution unit.
  • Present if test coverage can't be obtained for each test in the execution unit individually.
testsArray<TestInfo>optional
  • See section for TestInfo.
  • The set of tests that are executed in this execution unit.
  • In the UI these tests will be shown in the hierarchy below the execution unit.

# Sample File

The following JSON sample file combines the necessary data described above.

{
  "version": 2,
  "partial": true,
  "files": [
    {
      "path": "example_project/src/com/example/project/Calculator.java"
    },
    {
      "path": "example_project/test-src/com/example/project/CalculatorTest.java"
    }
  ],
  "tests": [
    {
      "uniformPath": "com/example/project/CalculatorTest/testAddition()",
      "duration": 1.5,
      "result": "PASSED",
      "coverage": {
        "0": "20-24,26,27,29",
        "1": "17-22"
      }
    },
    {
      "uniformPath": "com/example/project/CalculatorTest/testSubtraction()",
      "duration": 2.0,
      "result": "PASSED",
      "coverage": {
        "0": "20-24,26,27,29",
        "1": "17-22"
      }
    }
  ],
  "executionUnits": [
    {
      "uniformPath": "unit-tests/calculator-tests",
      "hash": "74E9F552F9543754373D4F07EB9D190E746A574A",
      "coverage": {
        "0": "20-24,27,29",
        "1": "19-25"
      },
      "tests": [
        {
          "uniformPath": "com/example/project/CalculatorTest/testMultiplication()",
          "duration": 3.0,
          "result": "PASSED"
        },
        {
          "uniformPath": "com/example/project/CalculatorTest/testDivision()",
          "duration": 4.0,
          "result": "PASSED"
        }
      ]
    }
  ]
}