Teamscale Documentation

# Recording Test Coverage for JavaScript Applications

This how-to describes how test coverage information can be recorded for a JavaScript application using the Teamscale JavaScript Profiler.

The outlined approach is particularly suited for scenarios where the system under test is deployed to a server and tests are running against that server. This might either happen via manual tests or also by automated UI tests.
It is also suited for legacy systems that use a testing approach with no explicit means to collect coverage information. For tests with a less complicated setup, for example, JavaScript unit tests, there are often simpler solutions which are discussed under alternatives.

Public Beta

The Teamscale JavaScript Profiler is still in the public beta phase. Your development and testing environment might not yet be fully supported by this approach. Please contact our support (support@teamscale.com) in case you encounter any issues.

The profiler consists of two major components: the instrumenter and the collector. The instrumenter adds statements to the code that signal reaching a particular code line when running it in the browser. The obtained coverage is aggregated in the Web browser and sent to a collecting server (the collector) once a second. Besides the coverage information, also the source maps of the code in the browser are sent to the collector once. The collector uses the source map to map the coverage information back to the original code and builds a coverage report that can be handed over to Teamscale. Teamscale uses the coverage information, for example, for Test Gap analysis (opens new window).

An overview of the components of the Teamscale JavaScript Profiler and their interactions is given in the following illustration:

Overview

# Prerequisites

To use the approach, a number of prerequisites have to be in place.

The instrumented code must be executed in a (possibly headless) browser environment that supports at least ECMAScript 2015. Furthermore, we require that a DOM and WebSockets are available in that execution environment. In other words, the approach supports Edge >= v79, Firefox >= v54, Chrome >= v51, and Safari >= v10. Instrumented applications cannot be executed in NodeJS.

To run the components of the profiler, NodeJS in at least version 14 is needed.

# Preparing your Application

Before we can instrument the application for sending coverage information to the coverage collector, the application has to be prepared: (1) source maps are needed to map back to the original code, (2) and the content security policy has to be adjusted to allow for sending the coverage information to the collector.

# Source Maps

The code that is executed in the browser often does not correspond to the code written by the developers. It can be the result of several transformation steps, for example, compilation (transpilation) from other languages, source code minimization, or bundling.

The presence of source map files in the code of the test subject ensures that the tested code can be mapped back to the original. Depending on your build pipeline, a different approach must be chosen to add the source maps to the test subject's code bundle.

In the following we provide pointers to relevant configuration options for some of the popular tools used in context of JavaScript applications:

# Content Security Policy

To use this coverage collecting approach, the application's Cross-Origin Resource Sharing (CORS) has to be adjusted. The instrumented application sends coverage information via WebSockets to a collecting server. That is, communication via WebSockets must be allowed. Whether or not this is allowed is determined by the Content-Security-Policy attribute. This attribute is either part of the HTTP header sent by the Web server delivering the Web application, or by a corresponding HTML entry (opens new window). If the collecting server is running on the same machine as the browser, then communicating with localhost must be allowed by adding ws://localhost:* for connect-src, blob, and worker-src to the Content-Security-Policy header.

The following snippet shows the content security policy that has to be added for allowing accessing the collector at host <collectorHost> on port <port>:

connect-src 'self' ws://<collectorHost>:<port>;
script-src 'self' blob: ws://<collectorHost>:<port>;
worker-src 'self' blob: ws://<collectorHost>:<port>;

By not specifying a content security policy, everything would be allowed. This can also be specified explicitly, for testing environments:

default-src * data: blob: filesystem: about: ws: wss: 'unsafe-inline' 'unsafe-eval' 'unsafe-dynamic'; script-src * data: blob: 'unsafe-inline' 'unsafe-eval'; connect-src * data: blob: 'unsafe-inline'; img-src * data: blob: 'unsafe-inline'; frame-src * data: blob: ; style-src * data: blob: 'unsafe-inline'; font-src * data: blob: 'unsafe-inline';

The place to configure the content security policy depends on the backend framework that serves the frontend code. See, for example, the Spring documentation (opens new window) on that topic.

# Coverage Collection

Before we describe how the code has to be instrumented to produce and send coverage information, we now describe how to set up the coverage collector. The address of this collector is later instrumented into the code of the test subject.

# Installing and Running

The collector is available as a NodeJS package. The package is available with the name @teamscale/coverage-collector in the NodeJS package manager.

# Running using NPX

The collector can be installed and started using the npx command. The following command starts the collector on the default port 54678. The coverage will be dumped into the default folder ./coverage:

npx @teamscale/coverage-collector

# Running as Node Script

The package @teamscale/coverage-collector can be added as a development dependency to the package.json file. For example, by running npm install -D @teamscale/coverage-collector (or yarn add -D @teamscale/coverage-collector).

After installing the package it should be registered in the package.json and be available locally for being executed. Please check the NPM package registry (opens new window) for the latest version of the package regularly.

Now we have to start the collector before testing is done, and have to stop it after this process has been finished. For this, we propose to use the pm2 package—can be installed using npm install -D pm2 (or yarn add -D pm2). The usage of pm2 is illustrated by following scripts in a package.json (assuming that yarn is used):

"scripts": {
  "collector": "coverage-collector",
  "pretest": "npx pm2 delete CC; npx pm2 start npm --name CC -- run collector",
  "test": "jest",
  "posttest": "npx pm2 delete CC"
},

Please see the npmjs documentation (opens new window) for details on the pre and post scripts used in the above example.

ATTENTION

These scripts do not include an instrumentation step, which is mandatory for producing coverage information. Such a step will be introduced later in this how-to.

# Configuration

The collector has three parameters that are relevant for typical application scenarios.

# Collector Port

-p PORT, --port PORT  The port to receive coverage information on.

The port the collector is listening on for information from the JavaScript applications under test is configured with the parameter --port. By default, the collector will listen on port 54678. Please make sure that this port is accessible (allowed by firewalls) by all clients conducting tests.

# Coverage File

-f DUMP_TO_FOLDER, --dump-to-folder DUMP_TO_FOLDER  Target folder

The collector dumps coverage information to files in the Teamscale Simple Coverage format. By default, this file is written after the collector terminates and every 2 minutes (see below). Every dump creates a new timestamped file inside the folder provided with the parameter --dump-to-folder. By default, coverage information is written to the folder coverage in the current working directory.

# Dump Interval

-t DUMP_AFTER_MINS, --dump-after-mins DUMP_AFTER_MINS
                        Dump the coverage information to the target file every N minutes.

The collector can be configured to dump coverage information regularly after a configured time interval has elapsed. The parameter --dump-after-mins allows the user to specify the number of minutes after the information is dumped. The default is set to 2 minutes. To disable this feature you can set it to zero (0) minutes.

# Control API

The upload parameters of the coverage collector can be controlled and queried remotely via a REST API. This API is enabled using the command line parameter --enable-control-port. For example, starting the collector with --enable-control-port 9872 makes the API available on port 9872 via HTTP.

-c ENABLE_CONTROL_PORT, --enable-control-port ENABLE_CONTROL_PORT
                      Enables the remote control API on the specified port (<=0 means "disabled").

The following REST API methods are available:

  • [PUT] /commit Sets the commit to use for uploading to Teamscale. The commit must be in the request body in plain text in the format: branch:timestamp
  • [POST] /dump Instructs the coverage collector to dump the collected coverage.
  • [PUT] /message Sets the commit message to the string delivered in the request body in plain text. This message will be used for all follow-up report dumps (see --teamscale-message).
  • [PUT] /partition Sets the name of the partition name to the string delivered in the request body in plain text. This partition will be used for all followup report dumps (see --teamscale-partition). For reports that are not directly sent to Teamscale the generated report will contain the partition name as session ID.
  • [POST] /reset Instructs the coverage collector to reset the collected coverage. This will discard all coverage collected in the current session.
  • [PUT] /revision Sets the revision to use for uploading to Teamscale. The revision must be in the request body in plain text.

Note that neither authentication nor transport encryption are required to control the collector. In case this is a strict requirement of your organization, please setup a corresponding reverse proxy that establishes and ensures these properties.

# Instrumentation

Before the coverage collector can receive any coverage information from a JavaScript application, this application has to be instrumented to collect and send this coverage information. Our JavaScript instrumenter package can be used for this purpose.

# Installing and Running

The instrumenter is available as a NodeJS package with the name @teamscale/javascript-instrumenter.

We recommend npx to execute the instrumenter. For example, the following command is used to instrument an example app.

npx @teamscale/javascript-instrumenter \
    test/casestudies/angular-hero-app/dist/ \
    --collector localhost:54678 \
    --in-place \
    --include-origin 'src/app/**/*'

This command instructs the instrumenter to instrument the code in the target folder test/casestudies/angular-hero-app/dist/. The instrumentation is done in-place (--in-place), that is, existing files are replaced by their instrumented counterparts.

# Configuration

The instrumenter can be configured by several parameters. We discuss some of them in the following sub-sections.

# Collector

  -c COLLECTOR, --collector COLLECTOR
                        The collector (`host:port` or `wss://host:port/` or `ws://host:port/`) to send coverage information to.

The parameter --collector allows for specifying the collector to send the coverage information to. Either a pair of hostname and port, separated by colon, or an URL pointing to the collector must be provided, for example, test-env.company.com:54678 or wss://test-env.company.com:54678. Please note that the specified collector must be reachable from clients that run the instrumented app to be able to collect coverage.

# Instrumentation Includes and Excludes

The instrumenter determines whether to instrument a particular code fragment or not by using include/exclude patterns. An JavaScript application is typically deployed by first performing various transformation steps on the original source file, for example, transpiling it from TypeScript to JavaScript, and then combining it with all the dependencies to bundles to be deployed and executed. For collecting coverage, the bundle files are then instrumented.

We provide two types of patters for excluding code from being instrumented: based on the origin of the code and based on the bundles the code was combined into.

# Origin-based

-x EXCLUDE_ORIGIN(S), --exclude-origin EXCLUDE_ORIGIN
                       Glob pattern(s) of files in the source origin to not produce coverage for. Multiple patterns can be separated by space.

-k INCLUDE_ORIGIN(S), --include-origin INCLUDE_ORIGIN
                       Glob pattern(s) of files in the source origin to produce coverage for. Multiple patterns can be separated by space.

These patterns match file names found in the original (!) source code files. That is, the source map—which is assumed to be present for files to instrument—is used to check if an instrumentation should be performed.

# Bundle-based

-e [EXCLUDE_BUNDLE ...], --exclude-bundle [EXCLUDE_BUNDLE ...]
                      Glob pattern(s) of input (bundle) files to keep unchanged (to not instrument).

This pattern matches the name of the final bundle files passed to the instrumenter as inputs.

# Specifying Multiple Patterns

Multiple include and exclude patterns should be separated by space. Exclude patterns have precedence over include patterns. For example, the following configuration includes all *.js files inside src/app1 and src/app2 and its subdirectories, except for files ending with .bin.js and .log.js:

npx @teamscale/javascript-instrumenter --include-origin 'src/app1/**/*.js' 'src/app2/**/*.js' --exclude-origin 'src/**/*.log.js' 'src/**/*.bin.js'

# Target Path

The instrumenter can either replace existing files by their instrumented counterparts, or it can write the instrumented versions to a separate target path.

-i, --in-place        If set, the original files to instrument are replaced by their instrumented counterparts.

If set, the flag --in-place instructs the instrumenter to replace the un-instrumented input files by their instrumented counterparts. Please be careful when using this parameter: Make sure that important changes to your code were saved in a separate location before performing the instrumentation. Typically, the in-pace instrumentation is performed on a target directory of the build process. The original source code files should not be instrumented with the presented approach—only copies of them!

-o TO, --to TO        Path (directory or file name) to write the instrumented version to.

In case the in-place instrumentation is not used and files are written to a separate path, the parameter --to has to be used to specify the target path.

# Uploading Coverage for Inspection

When the code to be tested was instrumented and the collector is running, code coverage will be produced and collected when running the code. By default, the collector will write coverage files in the Teamscale Simple Coverage Format.

Whenever a testing process has been finished (for example, in the build pipeline), the coverage can be provided to Teamscale for being used, for example, for a Test Gap Analysis. This can be done by using the Teamscale Upload Tool (opens new window) or by using the REST API directly. More details can be found in the corresponding documentation.

# Direct Upload from the Collector to Teamscale

Our coverage collector can be configured to send the collected coverage directly to a Teamscale server.

The upload is enabled by setting the URL of the Teamscale server using parameter --teamcale-server-url, along with parameters that define the target project and commit of the upload:

  -u TEAMSCALE_SERVER_URL, --teamscale-server-url TEAMSCALE_SERVER_URL
                        Upload the coverage to the given Teamscale server URL, for example,
                        https://teamscale.dev.example.com:8080/production.
  --teamscale-access-token TEAMSCALE_ACCESS_TOKEN
                        The API key to use for uploading to Teamscale.
  --teamscale-project TEAMSCALE_PROJECT
                        The project ID to upload coverage to.
  --teamscale-user TEAMSCALE_USER
                        The user for uploading coverage to Teamscale.
  --teamscale-partition TEAMSCALE_PARTITION
                        The partition to upload coverage to.
  --teamscale-revision TEAMSCALE_REVISION
                        The revision (commit hash, version id) to upload coverage for.
  --teamscale-commit TEAMSCALE_COMMIT
                        The branch and timestamp to upload coverage for, separated by colon.
  --teamscale-repository TEAMSCALE_REPOSITORY
                        The repository to upload coverage for. Optional: Only needed when uploading via revision to a
                        project that has more than one connector.
  --teamscale-message TEAMSCALE_MESSAGE
                        The commit message shown within Teamscale for the coverage upload. Default is "JavaScript coverage
                        upload".

Please note that all of the listed command line parameters can also be set via environment variables, for example, instead of passing the parameter --teamscale-access-token you can define the environment variable TEAMSCALE_ACCESS_TOKEN, instead of the parameter --teamscale-server-url you can define the variable TEAMSCALE_SERVER_URL.

# Direct Upload from the Collector to Artifactory

Our coverage collector can be configured to send the collected coverage directly to Artifactory.

The upload is enabled by setting the URL of the Artifactory server using parameter --artifactory-server-url, along with parameters that define the target partition and commit of the upload:

  --artifactory-server-url ARTIFACTORY_SERVER_URL
                        The HTTP(S) url of the artifactory server to upload the reports to. The URL may include a subpath on the artifactory server, e.g. https://artifactory.acme.com/my-repo/my/subpath.
  --artifactory-access-token ARTIFACTORY_ACCESS_TOKEN
                        (Preferred alternative to `artifactory-user` and `artifactory-password`) The API key for artifactory from
                        a user with write access (c.f. [Artifactory Documentation](https://www.jfrog.com/confluence/display/JFROG/User+Profile#UserProfile-APIKey))
  --artifactory-user ARTIFACTORY_USER
                        (Required, when not using artifactory-access-token) The name of an artifactory user with write access.
  --artifactory-password ARTIFACTORY_PASSWORD
                        (Required, when not using artifactory-access-token) The password of the user.
  --teamscale-partition TEAMSCALE_PARTITION
                        The partition to upload coverage to.
  --teamscale-revision TEAMSCALE_REVISION
                        (Optional) The revision (commit hash, version id) to upload coverage for.
  --teamscale-commit TEAMSCALE_COMMIT
                        The branch and timestamp to upload coverage for, separated by colon.

Please note that all of the listed command line parameters can also be set via environment variables, for example, instead of passing the parameter --artifactory-access-token you can define the environment variable ARTIFACTORY_ACCESS_TOKEN, instead of the parameter --artifactory-server-url you can define the variable ARTIFACTORY_SERVER_URL.

# Troubleshooting

# Instrumenter Runs Out of Memory

In case the application to instrument is too big, the instrumenter might run out of memory. In this case, you can increase the memory available to NodeJS by setting parameter max-old-space-size in the NODE_OPTIONS environment variable.

We recommend to use the cross-env package for setting the NODE_OPTIONS in NodeJS environments. For example, cross-env NODE_OPTIONS='--max-old-space-size=8192' npx @teamscale/javascript-instrumenter will increase the memory limit to 8GB for the given instrumenter invocation.

# Instrumented App Is Slow

After instrumenting your application for recording coverage information, it might become significantly slower. One cause of this could be that not only the application code was instrumented, but also the code of the frameworks (for example, Angular or React) and other libraries.

We recommend to instrument only those fraction of the application for that you would like to collect coverage information for. See Instrumentation Includes and Excludes.

# Alternatives

We describe one approach to record test coverage of JavaScript applications here. However, depending on the development setup and testing approach, there are a number of alternatives available, for example, Cypress (opens new window) can dump coverage information from the V8 JavaScript engine. Another example is NYC (opens new window), which can be used for collecting coverage of NodeJS applications; also c8 (opens new window) is applicable in that context. For unit tests, established tools such as Jest (opens new window) can produce coverage reports.

← Recording Test Coverage for Manual Tests →