Teamscale Docs

Appearance

Command-Line Client for Build (teamscale-build) ​

teamscale-build is a native command-line tool that supports typical use cases as faced on continuous integration servers and in CI/CD pipelines. The tool currently supports:

  • compressing coverage reports from common formats to a single compact Teamscale-specific report. This reduces the database size of Teamscale, the size of Teamscale backups and improves analysis performance.
  • converting multiple test-specific coverage reports to a single Testwise Coverage report. This is useful if your tooling can produce one coverage report per test case but not our custom Testwise Coverage JSON format.
  • extracting per-symbol binary size metrics for ELF and Mach-O binaries from a linker map file and a DWARF dump. The metrics can be uploaded to Teamscale as Custom Artifacts metrics.

How-To Guide

Our Tutorial describes how to use this tool for generating testwise coverage with a simple example.

Installation ​

Manual Installation ​

First Download the latest version for your operating system (Changelog):

Then extract the zip file and place it in a folder where your shell can pick it up (somewhere on your PATH).

Installation with Homebrew ​

On macOS, you can install teamscale-build with Homebrew using our custom tap.

First add the CQSE homebrew tap:

bash
brew tap cqse/teamscale https://homebrew.teamscale.com

Then install the teamscale-build CLI:

bash
brew install teamscale-build

Getting Started ​

The command-line client exposes its functionality as a set of commands. To list the available commands, use the --help command.

shell
teamscale-build --help

The various commands can be further configured by command-line options. In particular, you can use the help option, --help, built into all commands to list their respective options. For example:

shell
teamscale-build coverage testwise --help

Many options are common, while others are specific to a command and explained in the following alongside their respective command.

Commands ​

coverage compact ​

The coverage compact command converts multiple coverage reports of one format to a single Teamscale Compact coverage report. This format is a Teamscale-specific coverage format that allows to upload coverage information in a compressed format to reduce the amount of data processed by the Teamscale instance. If you are unfamiliar with the Compact coverage format, you might want to read the specification first. An example of the command would be:

shell
teamscale-build coverage compact -i /tmp/project -o /tmp/compact-coverage.json -f JACOCO

Command-specific options ​

There exists the following command-specific option:

  • -f/--format (required): The coverage format, must be one of the supported formats.
  • -u/--include-uncovered-lines (optional): Flag if uncovered lines should be included in the output. This is only relevant information for preprocessor languages like C/C++/C#. Default is false. You will have to enable the feature toggle -Dcom.teamscale.feature-toggle.use-coverable-lines-from-coverage-reports=true on your Teamscale instance to let Teamscale make use of this data. By default, Teamscale will ignore that information and determine the coverable lines heuristically.

Uploading the generated report ​

Use our upload tool teamscale-upload to upload the generated report to your Teamscale instance and specify the format TEAMSCALE_COMPACT_COVERAGE.

shell
teamscale-build coverage compact -i /tmp/project -o /tmp/compact-coverage.json -f JACOCO

TEAMSCALE_ACCESS_KEY=my-access-key teamscale-upload --server "$TEAMSCALE_URL" --project my-project --user ci --partition "Unit Tests" --format TEAMSCALE_COMPACT_COVERAGE /tmp/compact-coverage.json

coverage testwise ​

The coverage testwise command converts multiple coverage reports of one format to a single Teamscale Testwise Coverage report. If you are unfamiliar with the Testwise Coverage format, you might want to read the specification first. An example of the command would be:

shell
teamscale-build coverage testwise -i /tmp/project -o /tmp/testwise-coverage.json

This scans the given directory recursively for metadata.json files that point to the coverage files that should be processed, and outputs a merged Testwise Coverage report to the given json file.

Input path structure ​

The provided input paths (given via -i) will be traversed recursively for files named metadata.json. This metadata.json is expected to contain a testInfo object with the test's path, duration and result. A metadata.json might look like this:

{ "testInfo": 
  {
    "uniformPath": "my/cool/Test1", 
    "result": "PASSED",
    "duration": 1 
  }
 }

The given fields are:

  • uniformPath(String): Represents the test name that will be shown in Teamscale
  • result(String, default PASSED): The test execution result. Possible values for result include PASSED, IGNORED, SKIPPED, FAILURE, ERROR, and INCONCLUSIVE. See the Testwise Coverage specification for details on the semantics of those values.
  • duration(double, default 1): Duration of the test execution interpreted as seconds For a full reference of supported fields see TestInfo.

teamscale-build will look for test coverage to associate with the test next to the metadata.json. The coverage must be placed in directories named after the report formats (see supported formats) contained in them. If the folders contain multiple reports they will be merged.

An example directory schema be:

root
  arbitarily/many/subfolders
    testcase-folder1
      metadata.json
      JACOCO
        report1.xml
        report2.xml
        ...
    testcase-folder2
      metadata.json
      SIMPLE
        simple-coverage.txt
    ...

Where root is provided via the -i option. In this example, the coverage information from report1.xml and report2.xml would be mapped to the information provided in the metadata.json of testcase-folder1 (JaCoCo coverage) and simple-coverage.txt will be linked to the metadata.json of testcase-folder2 (SIMPLE coverage).

Command-specific options ​

The following command-specific options are available:

  • -c/--complete (optional): By default, the generated Testwise Coverage report is flagged as "partial", meaning Testwise Coverage in Teamscale for tests that are not included in this report is kept unchanged. If instead you’d like all tests that are not part of the generated report to be removed from Teamscale (e.g. because the tests were deleted meanwhile), you can mark the generated report as complete with this flag.
  • -u/--include-coverable-lines: (optional): Determines whether coverable lines should be included in the output. This is only relevant information for languages with a pre-processor like C/C++/C#. Default is false. You will have to enable the feature toggle -Dcom.teamscale.feature-toggle.use-coverable-lines-from-coverage-reports=true on your Teamscale instance to let Teamscale make use of this data. By default, Teamscale will ignore that information and determine the coverable lines heuristically.
  • -m/--method-accurate (optional): Determines whether the generated Testwise Coverage report will only contain method accurate coverage. Teamscale reports methods covered by method-accurate coverage as covered (for example in a Test Gap Treemap), but will not show any of their lines as covered in the Code File View. This flag is required to create a Testwise Coverage report for coverage in BULLSEYE format. Alternatively, use the coverage raw-testwise command.
  • -s/--subtract (optional): Directory containing the metadata.json of the test whose coverage should be subtracted from the coverage of the other tests. This option only works for coverage in BULLSEYE format and for exactly one report to subtract.

Subtracting test coverage

If your system contains a "background noise" of for example functions being called while the system is idling it can be useful to capture their coverage in a test case, that only consists of test setup and teardown, and to subtract the captured coverage from all other test cases. Otherwise, the coverage would impact the precision of Teamscale's test selection approaches, for example by suggesting more test cases than those that are actually testing a given functionality.

coverage raw-testwise ​

The coverage raw-testwise command converts multiple BULLSEYE coverage reports to a single Raw Testwise Coverage report. The Raw Testwise Coverage report is very similar to the output of the coverage testwise command. BULLSEYE reports however can only be converted to method-accurate non-raw Testwise Coverage reports via the coverage testwise ‑‑method-accurate command, since the original source code is needed for the conversion. Thus, to create Testwise Coverage that is not only method-accurate, this command packages the original reports into a single file that can be uploaded to Teamscale where the conversion happens.

Size of Raw Testwise Coverage reports

These Raw Testwise Coverage reports can become very big, and, thus, very slow to process. Teamscale rejects uploads of reports of more than 2 GB in size.

An example of the command would be:

shell
teamscale-build coverage raw-testwise -i /tmp/project -o /tmp/raw-testwise-coverage.json

For the upload you need to specify the RAW_TESTWISE format, which is available since Teamscale 9.2.

Input path structure ​

The input path and directory schema is the same as in the testwise command. The raw-testwise command supports any of Teamscale's supported coverage formats, but its main use-case remains the BULLSEYE format and the non-raw testwise command should be preferred if possible.

Command-specific options ​

The following command-specific options are available:

  • -c/--complete (optional): By default, the generated Raw Testwise Coverage report is flagged as "partial", meaning Testwise Coverage in Teamscale for tests that are not included in this report is kept unchanged. If instead you’d like all tests that are not part of the generated report to be removed from Teamscale (e.g. because the tests were deleted meanwhile), you can mark the generated report as complete with this flag.
  • -s/--subtract (optional): Directory containing the metadata.json of the test whose coverage should be subtracted from the coverage of the other tests. This option only works for coverage in BULLSEYE format and for exactly one report to subtract.
  • --include-coverable-functions (optional): Keep the coverable, but uncovered functions, which, by default, are removed from BULLSEYE coverage reports. Setting this flag results in larger report files, which are less performant to process in Teamscale. Unless you activated the feature toggle com.teamscale.feature-toggle.use-coverable-lines-from-coverage-reports in Teamscale to overwrite coverable lines calculated by it, Teamscale ignores these nodes by default.

binarysize extract ​

The binarysize extract command extracts the size of every symbol of a linked binary into a single non-code metrics report. Uploaded to Teamscale as Custom Artifacts metrics, this report shows which source files, functions and variables make up the size of the binary. Both ELF and Mach-O binaries are supported, the format is determined automatically from the given map file. Support for ELF binaries requires teamscale-build 0.4.0 or later.

An example of the command would be:

shell
teamscale-build binarysize extract -m /tmp/build/app.map -d /tmp/build/app.dwarfdump -o /tmp/binary-size.json

Input files ​

This command needs two input files, which need to be generated when building the binary.

The linker writes the map file when linking the binary:

shell
g++ -g -Wl,-Map=app.map -o app main.cpp

The debug information must be dumped to a DWARF file with llvm-dwarfdump:

shell
llvm-dwarfdump --debug-info --debug-addr app > app.dwarfdump

Dumps of other tools, such as objdump --dwarf, are rejected. --debug-info and --debug-addr restrict the dump to the sections that matter for the analysis. Setting --all includes all relevant section but is not recommended, as it can result in very large files.

Code of libraries that were built without debug information ends up in the report without source attribution.

Report contents ​

The report contains one entry per symbol. The entry paths have the following structure:

[<metric path prefix>/]<Symbols|Paddings|Unmapped>/<segment>[/<section>]/<source file>/<symbol>
  • Symbols contains the symbols of the binary.
  • Paddings contains bytes that belong to a section, but to no symbol, for example alignment bytes.
  • Unmapped contains symbols without debug information, for example the startup code of the toolchain. This currently applies to ELF binaries only.

Functions that the compiler inlined get their own entries. Their size is attributed to the inlined function and subtracted from the function it was inlined into.

Each entry carries up to two metrics. size is the number of bytes that the symbol occupies in the binary file, memorySize the number of bytes it occupies in memory at run time. For ELF binaries, sections that occupy memory but no space in the file, such as .bss, report only memorySize, and sections that are not loaded into memory, such as .debug_info, report only size. For Mach-O binaries, only size is reported.

Upload the report with the format GENERIC_NON_CODE, as described in How to Upload External Analysis Data. The metrics then show up in the Metrics perspective under Custom Artifacts.

Register the metrics in Teamscale

Since Teamscale 2026.7, size and memorySize are part of the external metrics schema by default, including on instances that were upgraded from an earlier version. On older versions, and whenever you choose custom metric ids via -t or -f, you have to add the metrics to the schema yourself. Metrics that are missing from the schema are dropped silently when the report is uploaded.

Command-specific options ​

The following command-specific options are available:

  • -m/--map: Path of the linker map file.
  • -d/--dwarf: Path of the DWARF dump.
  • -t/--size-metric (optional): Id of the metric for the size in the binary file. Default is size.
  • -f/--memory-metric (optional): Id of the metric for the memory footprint. Default is memorySize.
  • -p/--path-extraction-pattern (optional): Java regex containing exactly one group that determines which part of the source paths is kept, for example .*/(src/.*). Can be used multiple times. By default, the whole path is kept. Shortening the paths to those used in your Teamscale project lets the report entries link to the corresponding Code File View.
  • --include-path-pattern/--exclude-path-pattern (optional): Java regexes determining which entries end up in the report. An entry is kept if any include pattern matches and no exclude pattern matches. Both options can be used multiple times. Note that the path of a function contains its signature.
  • --metric-path-prefix (optional): Path prefix added to all entry paths, for example to keep the data of several binaries apart within one project.
  • --disable-inlined-function-handling (optional): Do not create separate entries for inlined functions and leave their size with the function they were inlined into.
  • --first-ignored-segment-pattern (optional): Java regex matching the name of a map file segment. The matching segment and all segments after it are ignored. By default, no segments are ignored.

If the sizes in the report do not add up to the sizes in the map file, the command reports the difference and exits with code 64. The report is written in that case, but its size attribution is incomplete.

Common Options ​

Many command-line options are shared between the commands offered by teamscale-build. These are described in the following.

Input path ​

Option name: -i or --input (required)

Path of a report file or a directory containing report files that should be parsed. Can be used multiple times within the same command. All reports must have the same coverage format. This option belongs to the coverage commands. binarysize extract reads its input from -m and -d instead.

Command-Specific Behavior

For coverage compact, a given directory is traversed recursively without a depth limit and every file in it is parsed as the format given by -f. Put nothing but reports of that one format below this directory, as any other file makes the command fail with a parse error. The coverage testwise and coverage raw-testwise commands traverse the input path differently, as described in Input path structure.

Input coverage format ​

Option name: -f or --format (required for coverage compact)

The report format of input coverage reports (case-insensitive). Can only be used once within the same command.

Available for coverage compact Only

This option exists for the coverage compact command only. The coverage testwise and coverage raw-testwise commands take the format of each report from the directory that contains it, as described in Input path structure.

Supported formats are

Teamscale-supported formats

The supported formats are a subset of all Teamscale-supported formats.

BULLSEYE Coverage Format

The BULLSEYE (BullseyeCoverage - C/C++) coverage format is only supported by the coverage testwise command with the ‑‑method-accurate flag and by the coverage raw-testwise command.

Output path ​

Option name: -o or --output-file (required)

Path of the generated report file. In case a file exists at this path, the file will not be overwritten, but instead, the command fails.

Logging ​

Option name: --stacktrace

In case any problems arise, or you need to report an issue to our support, you can log trace information by using the --stacktrace option.