Testwise Coverage (Version 1)

WARNING

Version 1 has some limitations that are resolved in version 2.

These include:

• Updating coverage information for a subset of tests is complicated
• No support information about the coverable lines as determined by the test coverage tool used to record it (needed mostly for C/C++ code)
• No way of including information about execution units which contain sets of tests that should or must be executed together

• Report Format
• Sample File for Testwise Coverage

Report Format

Testwise Coverage is a combined JSON-based report format that holds various test related data (see sample below). Its basis is a list of all tests that are available for execution as one unit (e.g. UI Tests). A test is identified by a unique string (uniformPath). The full list allows Teamscale to determine added and deleted tests. Each of the test entries can additionally contain execution information (duration, result, message), coverage information (paths) and additional test metadata (content). The following overview provides more details on each of the available fields:

uniformPath	Unique name of the test case by using a path like hierarchical description, which can be shown in the UI.
content	Identifies the content of the test specification. This can be used to indicate that the specification of a test has changed and therefore should be re-executed. The value can be an arbitrary string value e.g., a hash of the test specification or a revision number of the test.
duration	The time it took to execute the test in seconds.
result	Describes the result of the test execution. Can be one of: PASSED Test execution was successful. IGNORED The test is currently marked as "do not execute" (e.g. JUnit @Ignore or @Disabled). SKIPPED Caused by a failing assumption. FAILURE Caused by a failing assertion. ERROR Caused by an error during test execution (e.g., exception thrown). Tests that did not pass previously are always selected for test execution.
message	Error message and stacktrace or skip reason.
paths	The covered lines grouped by source file and path.

Sample File for Testwise Coverage

The following JSON sample file combines the necessary data described above.

{
  "tests": [
    {
      "uniformPath": "com/example/project/JUnit4Test/notExecutedTest()"
    },
    {
      "uniformPath": "com/example/project/JUnit4Test/systemTest()",
      "duration": 0.025,
      "result": "PASSED",
      "content": "43d743a8ef4389bc5",
      "paths": [
        {
          "path": "com/example/project",
          "files": [
            {
              "fileName": "Calculator.java",
              "coveredLines": "20-24,26,27,29"
            },
            {
              "fileName": "JUnit4Test.java",
              "coveredLines": "26-28"
            }
          ]
        }
      ]
    },
    {
      "uniformPath": "com/example/project/JUnit5Test/failedTest()",
      "duration": 0.045,
      "result": "ERROR",
      "message": "Some error"
    },
    {
      "uniformPath": "com/example/project/JUnit5Test/skippedTest()",
      "result": "SKIPPED"
    }
  ]
}