Testwise Coverage

• Report Format
  • Preliminaries
  • TestwiseCoverageReport
  • FileInfo
  • TestInfo
  • ExecutionUnit
• Sample File

Report Format

Testwise Coverage is a Teamscale-specific coverage format that allows to express coverage information on a per test level.

Versions

There are two versions of this report format. Version 2 is supported in Teamscale 7.8 and later. Version 1 is still supported for backwards compatibility. This document describes the version 2 of the Testwise Coverage report format. You can still find the documentation of the version 1 format here.

Preliminaries

• Uniform path
  • File path like string that represents a hierarchy. Parts of the hierarchy are delimited by slashes.
• Line range string
  • String containing line ranges and individual lines separated by commas.
  • Compact representation of a set of lines of code.
  • Example: "10-12,13,22-27,40"

TestwiseCoverageReport

The top level JSON object is the TestwiseCoverageReport. It contains the following attributes:

version	int	required	The version for the format described here should be 2. Version number for the Testwise Coverage report version. If not given the parser will fall back to version 1 of the Testwise Coverage format.
partial	bool	optional	If set to true the set of tests contained in the report don't represent the full set of tests. These tests are added or updated in Teamscale, but no tests that are missing in the report will be deleted. If not provided defaults to false.
files	Array<FileInfo>	required	See section for FileInfo. FileInfos are referenced elsewhere in the report by their index within the array.
executionUnits	Array<ExecutionUnit>	optional	See section for ExecutionUnit. Defaults to the empty array if omitted.
tests	Array<TestInfo>	optional	See section for TestInfo. Defaults to the empty array if omitted.

FileInfo

Information about a covered file. It contains the following attributes.

path	string	required	Code path within the repository. Ideally the exact relative path from the repository root. Otherwise, the path will be resolved to the longest matching suffix in Teamscale.
coverableLines	string	optional	Line range string for the lines in the source code that are considered coverable by the coverage tool, i.e. are reported with execution counts =0 or >0. If omitted Teamscale will determine coverable lines heuristically based on source code.

TestInfo

uniformPath	string	required	A unique hierarchical identifier of the test with / as separator. If this is a parameterized test the test may occur multiple times with different parameters. E.g. Java package name / test class name / test method name
parameters	Array<{value:string, name?:string}>	optional	A list of parameters with which the test has been executed. The value describes the parameters value. The name is optional and denotes the name of the parameter. The order of the parameters in the list is significant.
hash	string	optional	A hash determining the version of the test implementation. E.g. SHA-1 hash of a test description XML or a revision number of the test specification. If the value changes the test is considered as impacted.
duration	float	required	The duration in seconds it took to execute the test.
result	string	required	Describes the result of the test execution. Can be one of: PASSED Test execution was successful. IGNORED The test is currently marked as "do not execute" (e.g. JUnit @Ignore or @Disabled). SKIPPED Caused by a failing assumption. INCONCLUSIVE Test assertions cannot be verified. FAILURE Caused by a failing assertion. ERROR Caused by an error during test execution (e.g., exception thrown). Tests that did not pass previously are always selected for test execution.
message	string	optional	Error message and stacktrace or skip reason.
coverage	Map<string,string>	optional	A map of strings to line range strings. The keys are the integer indexes in string form of the FileInfo objects in the top-level files array. The values are the line range strings representing the covered lines.
associatedSpecItems	List<string>	optional	A list of specification items that this test verifies. An ID can be specified using a combination of requirements tracing connector name and specification item ID (e.g. req-connector|spec-item-id) or using a format matching the Specification item ID pattern option if this option has been set in the respective connector.

ExecutionUnit

Represents a set of tests that is executed together. An execution unit in the Testwise Coverage report can either contain a subset or all of the tests that are executed as one unit. Examples include test binaries built for C/C++ code or test Jar files for Java code. During the build process, all tests included in an execution unit will be executed at once to avoid overhead. It contains the following attributes.

uniformPath	string	required	A unique hierarchical identifier of the test with / as separator. E.g. unique path to executable or Jar file.
hash	string	optional	A hash determining the version of the execution unit. E.g. SHA-1 hash of test resources. If the value changes the execution unit is considered to be impacted.
duration	float	required	The duration in seconds it took to execute the execution unit.
result	string	required	The overall result of the execution of the execution unit. Can be one of: PASSED Execution was successful. IGNORED The execution unit is currently ignored. SKIPPED Caused by a failing assumption. INCONCLUSIVE Test assertions cannot be verified. FAILURE Caused by a failing assertion. ERROR Caused by an error during test execution (e.g., exception thrown). May differ from the combined result of tests contained in the execution unit. If the value changes the execution unit is considered to be impacted. Execution units that did not pass previously are always selected for execution in the context of Test Impact Analysis.
coverage	Map<string,string>	required	A map of strings to line range strings. The keys are the integer indexes in string form of the FileInfo objects in the top-level files array. The values are the line range strings representing the covered lines. The coverage produced by executing all the tests in the execution unit. Present if test coverage can't be obtained for each test in the execution unit individually.
tests	Array<TestInfo>	optional	See section for TestInfo. The set of tests that are executed in this execution unit. In the UI these tests will be shown in the hierarchy below the execution unit.

Sample File

The following JSON sample file combines the necessary data described above.

{
  "version": 2,
  "partial": true,
  "files": [
    {
      "path": "example_project/src/com/example/project/Calculator.java"
    },
    {
      "path": "example_project/test-src/com/example/project/CalculatorTest.java"
    }
  ],
  "tests": [
    {
      "uniformPath": "com/example/project/CalculatorTest/testAddition()",
      "duration": 1.5,
      "result": "PASSED",
      "coverage": {
        "0": "20-24,26,27,29",
        "1": "17-22"
      }
    },
    {
      "uniformPath": "com/example/project/CalculatorTest/testSubtraction()",
      "duration": 2.0,
      "result": "PASSED",
      "coverage": {
        "0": "20-24,26,27,29",
        "1": "17-22"
      }
    }
  ],
  "executionUnits": [
    {
      "uniformPath": "unit-tests/calculator-tests",
      "hash": "74E9F552F9543754373D4F07EB9D190E746A574A",
      "coverage": {
        "0": "20-24,27,29",
        "1": "19-25"
      },
      "tests": [
        {
          "uniformPath": "com/example/project/CalculatorTest/testMultiplication()",
          "duration": 3.0,
          "result": "PASSED"
        },
        {
          "uniformPath": "com/example/project/CalculatorTest/testDivision()",
          "duration": 4.0,
          "result": "PASSED",
          "associatedSpecItems": [
            "requirements-connector1|DP-3",
            "requirements-connector1|DP-4"
          ]
        }
      ]
    }
  ]
}