---
url: /howto/uploading-external-data.md
---

# How to Upload External Analysis Data to Teamscale

It is possible to upload external analysis data to Teamscale, either via the REST API or via repository connectors.
External analysis data include test coverage reports, test execution results, custom findings, custom metrics and more.

:::tip Uploading Vulnerability Reports
Vulnerability reports (such as SBOMs) are also external data, but they are uploaded with the `vulnerability-report` command of `teamscale-upload` and through a dedicated endpoint rather than the `external-analysis` sessions described here.
See [How to Upload Vulnerability Reports](../uploading-vulnerability-reports/) for details.
:::

:::tip Uploading test coverage from a CI pipeline?
Follow our [best practices for coverage uploads](../../introduction/best-practices/external-coverage-uploads/) to keep your Teamscale instance fast.
:::

\[\[toc]]

## Upload via Web UI

For testing and debugging purposes it can sometimes be helpful to [manually upload reports via the Web UI](../../reference/ui/project/project/#manual-report-upload).

## Upload via Command Line (`teamscale-upload`) {#upload-via-command-line}

With [our command line utility `teamscale-upload`](https://github.com/cqse/teamscale-upload/releases/latest) for Windows, Linux and macOS you can easily upload external analysis reports to Teamscale.

The CQSE website offers the **latest version** of the `teamscale-upload` utility for download.

* **Windows (x86):**
* **Linux (x86):**
* **macOS (ARM):**
* **macOS (x86):**

:::tip Homebrew on macOS
On macOS, the preferred way to install `teamscale-upload` is via our [Homebrew tap](https://github.com/cqse/teamscale-upload#installation-with-homebrew).
:::

The distribution contains the teamscale-upload tool and a corresponding Java execution environment (JVM) packed into a zip archive.
To use the tool unpack the archive and call the executable (`teamscale-upload/bin/teamscale-upload`).

An invocation may start with a command that says what to upload.
Naming no command or using the `report` command uploads external analysis reports.
Vulnerability reports use the separate `vulnerability-report` command, which is described in [How to Upload Vulnerability Reports](../uploading-vulnerability-reports/#upload-via-command-line).
Run `teamscale-upload/bin/teamscale-upload --help` to see all commands.

If you want to upload a report file from your CI pipeline, you can run:

::::tabs key:buildtool
\== Maven

```bash
teamscale-upload/bin/teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --accesskey <ACCESSKEY> --partition <PARTITION> --format jacoco **/target/site/jacoco/**.xml
```

\== Gradle

```bash
teamscale-upload/bin/teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --accesskey <ACCESSKEY> --partition <PARTITION> --format jacoco **/build/reports/jacoco/**.xml
```

:::tip Gradle Plugin
When using our Gradle Plugin it is not necessary to upload the report manually. There is a Gradle task for this. See [Upload via Gradle Plugin](#upload-via-gradle-plugin)
:::

\== Other

```bash
teamscale-upload/bin/teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --accesskey <ACCESSKEY> --partition <PARTITION> --format <FORMAT> <PATH/TO/REPORT/FILE>
```

::::

Every upload of external results needs to be linked to a commit in order to understand which exact versions were used during testing.
This command will automatically detect the version control commit to which the report belongs by querying your CI system (if it is supported, e.g. Jenkins, Azure DevOps, ...) or any Git or SVN checkout in the current working directory.
If you want to specify the commit manually see [here](#manually-specifying-the-commit-to-upload-to).

Choosing the correct format (`--format`) is crucial.
Otherwise, Teamscale cannot interpret the report file and will log errors to the Worker Log.
It must be one of the [formats supported by Teamscale](../../reference/upload-formats-and-samples/#supported-formats-for-upload).
Casing does not matter.

`teamscale-upload` will exit with code 0 if the upload was fully successful.
Any other exit code indicates a problem that you must address.
The problem is logged to the console.

Run `teamscale-upload report --help` to see all available options of the report upload.

:::warning Necessary Permissions in Teamscale
The user you supply on the command line must have the *Perform External Uploads* permission for the project to which you are uploading the reports.
For a production setup we recommend that you create a technical user in Teamscale and assign it the *Build* role for that project.
:::

:::warning Self-Signed Certificates
If you're accessing Teamscale via HTTPS and are signing the certificates yourself or via an internal CA, you can simplify the initial setup of this tool by using `--insecure`.
This will disable certificate validation.

Once you're successfully uploading your analysis results, you can re-enable validation and [provide the necessary certificates as a Java key store](../configuring-https/#creating-a-keystore) via `--trusted-keystore`.
:::

### Specifying the Teamscale access key

Pass the [access key](../../glossary/#access-key) via the environment variable `$TEAMSCALE_ACCESS_KEY` or via standard input by specifying `--accesskey -`:

```bash
$ export TEAMSCALE_ACCESS_KEY=<key>
$ teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --partition <PARTITION> --format <FORMAT> <PATH/TO/REPORT/FILE>
```

```bash
$ cat ~/accesskey.txt | teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --accesskey - --partition <PARTITION> --format <FORMAT> <PATH/TO/REPORT/FILE>
```

Passing the key directly via `--accesskey <key>` is not recommended.
The full command line is visible to anyone who can list the processes on the machine, and CI systems that echo their commands write it into the build log.
Use it only when neither of the options above is applicable.

### Manually specifying the commit to upload to

If you want to manually enter a commit you can do so with the `--commit` parameter.
It allows the following formats:

* The identifier of the supported VCS:
  * SHA Hash of the commit: e.g. `0842320ef195b2d9190ab32b2d24b4a77eb522b1`
  * SVN revision number: e.g. `1352456`
  * Team Foundation changeset ID: e.g. `1567`

It is also possible to use `--branch-and-timestamp`.
For details on this format please refer to the [glossary](../../glossary/#branch-and-timestamp).

### Uploading Multiple Report Files of the Same Format

If you need to upload more than one file of the same format, you can either specify more than one path on the command line or use wildcards:

* `*` matches any number of characters in the current directory or file name
* `**` matches any number of nested directories
* `?` matches any single character

:::warning Always Use Single Quotes Around Wildcard Strings
Since `*` and `?` are characters that are also interpreted by your shell, you must always use single quotes around paths with wildcards.
:::

E.g. `'path/**/report*.xml'` matches all of the following:

```
path/to/nested/report.xml
path/report.xml
path/report123.xml
```

### Uploading Multiple Files with Different Formats

If you need to upload more than one type of report, e.g. test coverage results and findings, you can use an input file.
In this file, you can define multiple formats and one or more patterns per format, e.g:

```ini
[jacoco]
jacoco1/**.xml
jacoco2/**.xml
[findbugs]
pattern1/**.findbugs.xml
pattern2/**.findbugs.xml
```

Use this file with the `--input` command line option:

```bash
teamscale-upload --server http://<TEAMSCALE_URL> --project <PROJECT> --user <USER> --accesskey <ACCESSKEY> --partition <PARTITION> --input <PATH/TO/INPUT/FILE>
```

## Upload via Docker Image

The same `teamscale-upload` utility is also available as a Docker image on [Docker Hub](https://hub.docker.com/r/cqse/teamscale-upload/tags).
This is usually the simplest option for CI pipelines that already run their build steps in containers, for example Azure DevOps, Jenkins or GitLab CI.

Mount your build workspace at `/workspace`, which is the working directory inside the image, and pass the `teamscale-upload` options after the image name:

```bash
docker pull cqse/teamscale-upload:latest
docker run --rm \
  -v "$PWD:/workspace:ro" \
  -e TEAMSCALE_ACCESS_KEY \
  cqse/teamscale-upload:latest \
  --server https://<TEAMSCALE_URL> --project <PROJECT> --user <USER> \
  --partition <PARTITION> --format jacoco 'build/reports/jacoco/**.xml'
```

The entrypoint of the image is `teamscale-upload` itself, so everything after the image name is passed to the tool.
This includes the command, so put `vulnerability-report` in front of the options to [upload a vulnerability report](../uploading-vulnerability-reports/#upload-via-docker-image).
See [Specifying the Teamscale access key](#specifying-the-teamscale-access-key) for the other ways to hand the access key to the tool.

:::tip Detecting the Commit Inside the Container
The image contains `git`, so mounting your checkout at `/workspace` is enough for the commit to be detected automatically.

When you run the image as a job container, the revision variable of your CI system is picked up as well (`CI_COMMIT_SHA` on GitLab CI, `GIT_COMMIT` on Jenkins or `Build.SourceVersion` on Azure DevOps).
A plain `docker run` step does not inherit these variables, so either forward one of them with `-e` or [specify the commit manually](#manually-specifying-the-commit-to-upload-to).
:::

:::warning Working Directory and File Permissions
CI systems that run the image as a job container mount their own workspace and set their own working directory, which overrides `/workspace`.
In that case, use the report paths of that CI system instead of paths below `/workspace`.

The container runs as the non-root user `teamscale`, so your report files must be readable for other users.
This is the case for the default permissions of most CI workspaces.
:::

Inside the image, `teamscale-upload` is also on the `PATH`.
This matters for CI systems that ignore the entrypoint and run their build steps in a shell instead, such as GitLab CI when you select the image via `image:`.
There, call `teamscale-upload` as a regular command.

## Upload via GitHub Action

If you are using GitHub as your CI system we recommend using teamscale-upload via GitHub Action which is available on the [GitHub Marketplace](https://github.com/marketplace/actions/teamscale-upload).
It provides the same options as teamscale-upload and can be used as follows:

```yaml
uses: 'cqse/teamscale-upload-action@v2.9.5'
    with:
      server: 'https://demo.teamscale.com'
      project: 'teamscale-upload'
      user: 'build'
      partition: 'Github Action > Linux Branch And Timestamp'
      accesskey: ${{ secrets.ACCESS_KEY }}
      format: 'SIMPLE'
      message: 'This is a test message.'
      files: 'test_resources/coverage.simple test_resources/coverage2.simple'
```

A more detailed description of the parameters can be found [here](https://github.com/cqse/teamscale-upload-action/blob/master/action.yml).

## Upload via REST API

If you cannot use the `teamscale-upload` utility, you can also manually upload files via our REST API.
We do recommend that you use `teamscale-upload` if possible since it includes helpful handling for many common errors, typos, etc.

The REST service `external-analysis/session/<SESSION_ID>/report` allows uploading results from analysis tools like [FindBugs/SpotBugs](https://spotbugs.github.io/) or different [coverage tools](../../reference/supported-technologies/test-coverage/).

### Prerequisites

In order to upload a report to Teamscale, the server which is running the
Teamscale instance must be accessible via HTTP(S).

As a first step, create a special user with the `Build` role which will be able to upload external analysis results to Teamscale.
For such a dedicated upload user, we recommend the **Access key only** authenticator, which you can select when editing the user under *Admin* > *Users*.
It disables interactive login to the Web UI, so the account can only authenticate via its [access key](../../glossary/#access-key) and it is excluded from Teamscale's contributor statistics.

### Uploading a single report

To upload a single external analysis report to Teamscale for the current HEAD commit (see below for [uploads for historic dates](#uploading-data-for-historic-dates-or-revisions)), issue the
following HTTP request:

* **HTTP method**: `POST`

* **Request URL**:

  ```bash
  http://<YOUR_TEAMSCALE_SERVER>:<PORT>/api/vTEAMSCALE_VERSION/projects/<TEAMSCALE_PROJECT_ID>/external-analysis/session/auto-create/report?format=<FORMAT>&partition=<PARTITION>&message=<MESSAGE>
  ```

  * `YOUR_TEAMSCALE_SERVER:PORT` is the address and port of the Teamscale instance.
  * `TEAMSCALE_PROJECT_ID` is the ID of the Teamscale project into which to store the coverage.
  * `FORMAT` is the report format of the uploaded file. It must be one of the [formats supported by Teamscale](../../reference/upload-formats-and-samples/#supported-formats-for-upload).
  * `PARTITION` is a logical name for the type of coverage. All
    coverage with the same partition will be grouped together. This,
    e.g., allows you to upload both coverage for your Java and your
    JavaScript tests without the results of one overriding those of
    the other. The name is descriptive and can be any arbitrary
    string.
  * `MESSAGE` is the message shown in the Activity perspective to the
    user. It should make clear that coverage was uploaded.
  * Optionally, `replaceExistingUploads=true` makes this upload replace earlier uploads for the same commit and
    partition instead of adding to them. This requires an external storage backend, see
    [Replacing Existing Uploads](#replacing-existing-uploads).

* **Body**: Form data. May contain any number of elements with name
  "report" and the content of a single result XML file.

* **Authentication**: HTTP Basic Authentication with the build or admin
  user and their Access Token (**not** their password!)

#### Sample Upload with Curl

```bash
curl --request POST --user bob:e434mikm35d --form "report=@coverage.xml" "http://teamscale-server:8080/api/vTEAMSCALE_VERSION/projects/myProject/external-analysis/session/auto-create/report?format=JACOCO&partition=Unit%20Tests&message=Unit%20Test%20Coverage"
```

:::tip Request Parameters
Details about the service's parameters can be best retrieved via the
online service documentation as [described here](../../reference/rest-api/#available-rest-services).
:::

:::warning Always Quote the URL
Please note that the URL must be enclosed in quotes.
This is required as most shells treat the `&` character specially and would thus truncate the URL.
If you are not sure you quoted the URL correctly, you can use `curl -v` to get verbose output from curl, including the actual URL it tries to contact.
:::

:::warning File encoding
If not explicitly specified, Teamscale will interpret the encoding of the provided file based on its BOM, or default to UTF-8. Change the `charset` of the `Content-Type` header when uploading a Teamscale report via `curl` to prevent parsing errors because of misinterpreted encodings.
:::

An example how to upload a report with a specified UTF-16 encoding:

```bash
curl --request POST --user bob:e434mikm35d --form "report=@coverage.xml;type=text/xml;charset=UTF-16" "http://teamscale-server:8080/api/vTEAMSCALE_VERSION/projects/myProject/external-analysis/session/auto-create/report?format=JACOCO&partition=Unit%20Tests&message=Unit%20Test%20Coverage"
```

Note the type definition `type=text/xml;charset=UTF-16` after the file name.

In addition to the [common pitfalls
when dealing with the REST API](../../reference/rest-api/#common-pitfalls), you need to check if the external tool is
activated in the project's analysis profile.

Furthermore, please refer to the [details for supported upload formats](../../reference/upload-formats-and-samples/).

### Uploading data for historic dates or revisions

Note that the uploaded results will be used for the current
HEAD commit in Teamscale on the main branch (e.g., latest commit on
master for Git, latest changeset on the TFS main branch or latest commit
on trunk for SVN). Should this not be the behavior you want, you can
override this by adding the following to the request's URL:

`&revision=<REVISION>`

where REVISION is, depending on your version control system, a:

* full Git SHA hash, or
* SVN revision number, or
* TFS changeset ID

:::warning Use only during session creation
Note that the `revision` parameter has to be supplied [when creating the session](#uploading-multiple-reports-at-once-sessions). It will **not** work for individual uploads.

Please refer to [interactive REST API documentation](../../reference/rest-api/) to view all parameters.
:::

Regardless of whether the revision has already been analyzed or not, Teamscale will store the external analysis data for the code with the revision.
When later it is analyzed, the upload is integrated with the revision into Teamscale.

If concerned project has a multi-connector setup, the VCS repository that
contains the revision for which an upload is meant, can be specified in
the request's URL using the repository identifier (see [connector options](../../reference/connector-options/#shared-connector-options))
for the project connector:

`&repository=<REPOSITORY_IDENTIFIER>`

Teamscale also supports uploading external analysis data to
timestamps. This can be achieved by adding the following to the
request's URL:

`&t=<TIMESTAMP>`

If you additionally need to specify the branch for the upload, you can use the `t` parameter as well:

`&t=<BRANCH>:<TIMESTAMP>`

#### Choosing between the `revision` and `t` parameters

Both parameters specify the commit to which your upload belongs, but they do not behave the same way.

The `revision` parameter links the upload to a commit that Teamscale resolves in one of the connected repositories.
The data is integrated with that commit once it has been analyzed, as described above, and no branch is created for it.

The `t=<BRANCH>:<TIMESTAMP>` parameter links the upload to a [branch and timestamp](../../glossary/#branch-and-timestamp) that you choose.
Teamscale imports the data at exactly that timestamp on that branch.
If the project uses an [external storage backend](#storing-analysis-results-in-an-external-storage-system), this also creates the branch if it is not yet known to Teamscale.

Prefer the `revision` parameter whenever your build knows the revision, so that your data stays linked to the exact code version it was produced from.
Use `t=<BRANCH>:<TIMESTAMP>` when you deliberately want the data stored at a specific point in time, for example when a manual test campaign runs over several days and each day's results belong to the day they were produced.

Note that the revision you specify is not kept as-is for every version control system.
If the project uses an external storage backend, a revision that is a plain number, such as an SVN revision number or a TFS changeset ID, is stored using the branch and timestamp it resolves to instead of the revision itself.
Such revision numbers only increase over time and cannot be assumed to be unique across repositories, and for these version control systems the branch and timestamp are the more reliable reference anyway, because their branch names are far more stable than in Git.

:::warning Uploading to a branch that contains no code
Report types that refer to specific code, most importantly test coverage, show no results if the branch you upload to contains no code, because there is no code to which the results can be assigned.
Report types that do not depend on code, such as non-code metrics, are imported normally.
A mistyped branch name therefore creates a branch that contains nothing but your uploads.
:::

### Uploading Multiple Reports At Once (Sessions)

If you want to upload multiple reports in a single logical commit, you
should use Teamscale's session concept. For session handling, the
service `external-analysis/session` is used. The following example uploads two
different reports as a single commit using *curl*. For more details, see
the [service documentation](../../reference/rest-api/#available-rest-services).

```bash
curl --request POST --user bob:e434mikm35d http://localhost:8080/api/vTEAMSCALE_VERSION/projects/my-project-id/external-analysis/session?message=myMessage&partition=myPartition
```

This command will return the session ID, in this case `Sdfb38cb0-c348-40fb-82f0-144bf0e41beb`.
Using this session, we can upload multiple reports.

```bash
curl --request POST --user bob:e434mikm35d --form "report=@./MyReport.xml" http://localhost:8080/api/vTEAMSCALE_VERSION/projects/myProject/external-analysis/session/Sdfb38cb0-c348-40fb-82f0-144bf0e41beb/report?format=FINDBUGS
```

```bash
curl --request POST --user bob:e434mikm35d --form "report=@./OtherReport.xml" http://localhost:8080/api/vTEAMSCALE_VERSION/projects/myProject/external-analysis/session/Sdfb38cb0-c348-40fb-82f0-144bf0e41beb/report?format=FXCOP
```

Finally, we commit the session using a POST request:

```bash
curl --request POST --user bob:e434mikm35d http://localhost:8080/api/vTEAMSCALE_VERSION/projects/my-project-id/external-analysis/session/Sdfb38cb0-c348-40fb-82f0-144bf0e41beb
```

## Upload via Gradle Plugin

With our [Teamscale Gradle Plugin](../../reference/integrations/gradle-plugin/) you can easily upload artifacts directly from Gradle without any additional tooling involved.
It can automate uploading arbitrary reports.
It has special support for JUnit, JaCoCo, [Compact Coverage](../../reference/upload-formats-and-samples/teamscale-compact-coverage/) and [Testwise Coverage](../../reference/upload-formats-and-samples/testwise-coverage/), which means that you can just point the plugin to the corresponding task that creates the report and it will take care of wiring up the report to the upload task.
This guide expects that you have already configured your test tasks.

:::tip Test Impact Analysis

For more in depth instructions on how to set up Testwise Coverage collection and Impacted Test execution please refer to our [tutorial](../../tutorial/tia-java/).
:::

You can apply the plugin by adding it to your plugins block of the top level project.

:::: tabs key:gradle\_lang
\== Groovy

```groovy
plugins {
    id "com.teamscale" version "35.0.2"
}
```

\== Kotlin DSL

```kotlin
plugins {
    id("com.teamscale") version "35.0.2"
}
```

::::

The `teamscale` extension allows you to configure where reports should be uploaded to:

:::: tabs key:gradle\_lang
\== Groovy

```groovy
teamscale {
    server {
        url = 'https://mycompany.teamscale.io/'
        project = 'test'
        userName = 'build'
        userAccessToken = property('teamscale.access-token')
    }
}
```

\== Kotlin DSL

```kotlin
teamscale {
    server {
        url = "https://mycompany.teamscale.io/"
        project = "test"
        userName = "build"
        userAccessToken = System.getProperty("teamscale.access-token")
    }
}
```

::::

The example assumes that you pass the Teamscale Access Token to Gradle via

```bash
gradle -Pteamscale.access-token=82l1jtkIx6xG7DDG34FLsKhejcHz1cMu ...
```

By default the plugin will detect the commit it needs to upload to from either the environment variables set in the CI pipelines or from the current Git checkout.
In case none of these work in your setup, you need to specify the commit hash manually:

:::: tabs key:gradle\_lang
\== Groovy

```groovy
teamscale {
    commit {
        revision = System.getenv('MY_ENV_COMMIT_SHA')
    }
}
```

\== Kotlin DSL

```kotlin
teamscale {
    commit {
        revision = System.getenv("MY_ENV_COMMIT_SHA")
    }
}
```

::::

The `TeamscaleUpload` task allows you to upload reports to Teamscale:

:::: tabs key:gradle\_lang
\== Groovy

```groovy
import com.teamscale.TeamscaleUpload

tasks.register('teamscaleTestUpload', TeamscaleUpload) {
    partition = "Unit Tests"
    from(tasks.test) // Test task
    from(tasks.jacocoTestReport) // JacocoReport task
}
```

\== Kotlin DSL

```kotlin
import com.teamscale.TeamscaleUpload

tasks.register<TeamscaleUpload>("teamscaleTestUpload") {
    partition = "Unit Tests"
    from(tasks.test) // Test task
    from(tasks.jacocoTestReport) // JacocoReport task
}
```

::::

`from(task)` supports tasks of the following types:

* `Test` will upload the resulting JUnit XML reports (if `reports.junitXml.required` is enabled) containing information about the tests that were executed, their durations and results. This does not include test coverage.
* `JacocoReport` will upload the resulting JaCoCo XML report. This contains test coverage, but no test execution results.
* `CompactCoverageReport` will upload the resulting [Compact Coverage](../../reference/upload-formats-and-samples/teamscale-compact-coverage/) report. Similar to the JaCoCo report, this report contains test coverage but with a smaller footprint that consumes less storage; it should be preferred over JaCoCo.
* `TestwiseCoverageReport` will upload the resulting [Testwise Coverage](../../reference/upload-formats-and-samples/testwise-coverage/) report. This contains test coverage and test execution information. In contrast to JaCoCo and Compact Coverage, Testwise Coverage holds test coverage per test case.

`addReport(format, reportFiles)` allows you to also upload any other reports (see [Upload Formats](../../reference/upload-formats-and-samples/))

You can run this as part of your CI Gradle command like this:

```bash
gradle --continue test teamscaleTestUpload
```

`--continue` tells Gradle to upload the reports even when the tests failed.

For more details have a look at the [plugin reference documentation](../../reference/integrations/gradle-plugin/).

## Upload via Maven Plugin

Analogously to the Gradle plugin, there is a [Teamscale Maven plugin](https://mvnrepository.com/artifact/com.teamscale/teamscale-maven-plugin) that can be used to upload JaCoCo reports to Teamscale.
It can be used to automatically upload Unit, Integration and Aggregate test coverage reports generated by the [JaCoCo Maven plugin](https://mvnrepository.com/artifact/org.jacoco/jacoco-maven-plugin) to your Teamscale instance.
This guide expects that you have already configured the [JaCoCo plugin](https://www.jacoco.org/jacoco/trunk/doc/maven.html).

:::tip Recording testwise coverage
The Teamscale Maven plugin can also be used to directly record and upload test coverage to Teamscale.
In addition, it is possible to record testwise coverage with it.
Please have a look at our [testwise coverage section](../providing-testwise-coverage/#maven-and-junit-5) for a more detailed example.
:::

The plugin offers a goal called `upload-coverage` which can be invoked after the JaCoCo test reports have been generated.

:::warning Check the JaCoCo configuration
Make sure that JaCoCo is configured to [generate XML reports](https://www.jacoco.org/jacoco/trunk/doc/report-mojo.html#formats).
:::

Configure where reports should be uploaded to by supplying the `<teamscaleUrl>` and `<projectId>`.
An example configuration could look like this

```xml
<plugin>
        <groupId>com.teamscale</groupId>
        <artifactId>teamscale-maven-plugin</artifactId>
        <version>34.2.2</version>
        <executions>
            <execution>
                <goals>
                    <goal>upload-coverage</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <teamscaleUrl>http://localhost:8080</teamscaleUrl>
            <projectId>my-project</projectId>
            <username>build</username>
            <accessToken>6lJKEvNHeTxGPhMAi4D84DWqzoSFL1p4</accessToken>
            <endCommit>master:HEAD</endCommit>
            <unitTestPartition>My Custom Unit Tests Partition</unitTestPartition>
        </configuration>
</plugin>
```

You can optionally specify an `endCommit`, if left out the currently checked out commit will be automatically detected from the project's git repository.
You can use `unitTestPartition` to define the partition to which unit tests should be uploaded to, similar to `integrationTestPartition` and `aggregateTestPartition`.

Invoking `mvn com.teamscale:teamscale-maven-plugin:upload-coverage` will collect all generated XML reports and upload them to each partition in a single session.
If you have a Maven project consisting of multiple submodules, you have to invoke the command in the root project.

## Uploading from Jenkins

For Jenkins users we also have a plugin that provides an "after build step" to do the upload.
For detailed instructions please see the [Jenkins plugin page](https://plugins.jenkins.io/teamscale-upload/).
If your pipeline runs its build steps inside containers instead of using the plugin, see [Upload via Docker Image](#upload-via-docker-image).

## Upload from Azure DevOps

A plugin is provided for directly uploading coverage from TFS / Azure DevOps.

In order to authenticate against Teamscale, first add a variable to your pipeline that contains a Teamscale [access key](../../glossary/#access-key).
When using the classic editor, you can add it on the "Variables" tab.

![In the classic editor you can use the "Variables" tab for adding variables](./teamscale-access-key-1.png)

When using a YAML pipeline, you have to open the pipeline for editing, then click on the "Variables" button.

![In the YAML editor you have to click on the "Variables" button in Edit mode for adding variables](./teamscale-access-key-2.png)

Change the variable type to "secret".

Add the task "Teamscale Report Uploader" to your pipeline.

![In the YAML editor you can search for the task name "Teamscale" and add the corresponding "Teamscale Report Uploader" task](./teamscale-report-uploader-task.png)

Fill the following parameters:

* Files: [Glob pattern](../../glossary/#glob-pattern) for the reports to be uploaded.
* Report Format ID: You can find all valid IDs on the [Supported Upload Formats and Samples](../../reference/upload-formats-and-samples/#supported-formats-for-upload) page.
  If you want to upload multiple report formats, you need to add one pipeline task per format.
* Teamscale URL: The URL under which Teamscale is accessible.
* Username: The Teamscale username for which you have generated the Teamscale access key.
* IDE Access Key: The secret variable you have added to the pipeline, e.g. `$(teamscaleAccessKey)`.
* Teamscale Project: The project ID of the corresponding Teamscale project. You can copy this from the URL when you open the project in Teamscale or find it in the *Projects* view.
* Partition (see [Glossary](../../glossary/#partition)): An arbitrary name to associate with the report uploads. Note that it's recommended to have a separate partition per pipeline and per report format, e.g. `My Pipeline - Cobertura`.
* Path to CodeCoverage.exe (optional): This must be defined if you want to upload a binary Visual Studio coverage file. The task will use this to convert the binary file to an XML file before the upload.
  For detailed instructions on how to convert the coverage files, see the [TFS / AzureDevOps Section](../setting-up-profiler-tga/cs/#codecoverage-exe).

The task will automatically resolve which revision it's running for, and will use this revision for the upload.

## Import via Artifactory / S3

External analysis data can be directly imported to Teamscale via the Artifactory or S3 repository connector.
The connector will extract the results from the repository, process the reports and store them in the configured partition.

### Layout Prerequisites

Before importing external analysis data to Teamscale via Artifactory / S3, please ensure the following:

* All relevant external results reports are available in Artifactory / S3.
* The external results reports are packed in archives (i.e. ZIP files).
* The folder structure used follows this pattern:
  ```
  / arbitrarily / many / directories / before / the / upload / root
     / 'uploads'
     / <branch name> 
     / <commit timestamp in ms (Java long)>[-<commit hash>]?
     / <upload partition> 
     / <artifact type (EReportFormat, case insensitive), e.g. 'jacoco'> 
     / none / or / some / intermediate / directories /
     / <Zip archive file>
  ```
  All profilers / upload tools provided with Teamscale follow this format automatically, making it easy to configure.
  If you use your own schema, make sure to adapt the extraction patterns accordingly.

### Configuration Steps

To import external results reports to a project:

1. Add an Artifactory / S3 connector to the project.
2. Configure the [Artifactory connector options](../../reference/connector-options/#artifactory-connector-options) so that Teamscale can extract the reports from Artifactory.
3. If you used the standard path format as specified in [Prerequisites](#layout-prerequisites), you can skip this step. Otherwise, configure the following options so that Teamscale can process the reports:
   * `Analysis report mapping`: Glob pattern used to map the reports inside the ZIP archives to their corresponding [report format](../../reference/upload-formats-and-samples/#supported-formats-for-upload).
     Example: `**/junit.xml -> JUNIT, **/jacoco.xml -> JACOCO`
   * `Partition Pattern`: Regular expression used to extract the partition from the path of the artifacts.
     The first non-null group is used as partition, e.g. `/coverage/([^/]+)/` would use the sub-folders of coverage as the partition names.
4. Save the project.

### Configuration Example

Below is the structure of the Artifactory containing the external results reports.

* `lcov-reports.zip` contains the LCOV coverage report (`lcov.info`).
* `findings-reports.zip` contains the external findings report for findings (`warnings.json`).

```
- MyRepository
|
| -> MyProjectName
  |
  | -> uploads
    |
    | -> master
      |
      | -> 1578650400000-4e8afe52737255ba40875a1af1fced1e08fc5316
        |
        | -> coverage
          |
          | -> LCOV
            |
            | -> lcov-reports.zip
        |
        | -> findings
          |
          | -> GENERIC_FINDINGS
            |
            | -> findings-reports.zip
      |
      | -> 1578651300000-70eb476d01abd0d877d064f57a6da5e14c677687
        |
        | -> coverage
          |
          | -> LCOV
            |
            | -> lcov-reports.zip
```

Accordingly, the Artifactory connector's configuration would look as follows:

![Screenshot of the Artifactory connector's configuration to upload external results](./artifactory-options.png)

And this would be the end result and how the uploaded results via Artifactory would appear in Teamscale:

![Screenshot of the uploaded external results via Artifactory](./artifactory-uploaded-results.png)

## Storing analysis results in an external storage system

:::tip

This section does not pertain to [imports from storage systems](#import-via-artifactory-s3) and is only concerned with direct uploads to Teamscale.
:::

By default, uploads of external analysis data over the REST API interface offered by Teamscale will be stored in Teamscale's internal storage.
However, Teamscale also provides the option to use an external storage system to store the data.
Some advantages of using an external storage backend are:

* Data can be shared over multiple projects without duplicating it, which is especially useful for shadow instances and project copies.
* The size of Teamscale backups is not impacted by external analysis data, as the data is pulled from the external storage during history analysis.
* Dedicated storage services can provide more elaborate data management than Teamscale. For example, automatically pruning old uploads is easier.
* External data might need to be persisted anyway. In this case, duplication is avoided.
* Teamscale can always access the full report data, leading to better analysis results in some rollback scenarios.

### Configuring Teamscale to Store External Analysis Data in S3

:::tip S3 server required

This requires an S3 server for storage, which is not shipped with Teamscale.
:::

:::tip Required permissions

This configuration can only be done by users with permissions to create external storage backends.
:::

1. Create an external account, containing the URL, user and API key. The account has to specify "multiple systems" (default) or "S3" to show up in the following configuration.
2. Navigate to `Admin -> Storage Backends`. This page contains a list of all configured external storage backends (internal storage is not displayed, since it is the default behavior).
3. Click `New external storage backend` to open the configuration dialog.

:::details Click here for a detailed explanation of every field in the dialog.

| Field Name                    | Description                                                                                                                                                                                          |
|-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Name                          | An arbitrary name to identify this storage backend. This can not be changed once the storage backend is created.                                                                                     |
| External Account              | Contains the credentials and URL to authenticate with S3. The selected S3 account needs at least creation and deletion permissions on the S3 server to allow uploading data.                         |
| Protocol                      | The protocol used by the storage system S3 in this case (other protocols - e.g. Artifactory - will be supported in the future).                                                                      |
| Repository or bucket          | In case of S3, this is the bucket name Teamscale should upload to.                                                                                                                                   |
| Upload Path Prefix (Optional) | By default, Teamscale will upload external data to the root of the selected S3 bucket. This option can be used to specify an optional subdirectory (relative to the S3 bucket root) where the uploads will be stored instead, e.g. `teamscale/external-analysis-uploads/`. |
:::

4. Once the external storage is configured, it still needs to be enabled to have any effect.
   There are two ways to enable an external storage: on a global basis, or on a project-by-project basis.
   1. By default, Teamscale stores uploads internally, along with the rest of its analysis data. To change this default
      storage location, instead, select one of the configured external storage backends as default. New projects will
      automatically pre-select the default. Old projects will only use the default after editing the project, including a
      reanalysis.

   2. Sometimes, you may want to migrate projects on a server one-by-one.
      To change the external storage backend of a project, navigate to the `Edit Project` page and select the storage
      backend from the dropdown.

:::danger

Changing the configuration of the selected storage, e.g., bucket name or path, may change the analysis history of the
respective projects, automatically triggering a (partial) re-analysis.
:::

:::warning Copying projects with external storage

The copy of a project with external storage backend will keep the mapping to the external storage backend of the
original project. Thus, it also contains the stored analysis results of the original project. To create a separate
external storage backend mapping for the copy, uncheck the "Copy all project data & share external storage data" checkbox. Note that this also omits
any external data which was stored in Teamscale before external storage was enabled. This *will also share future uploads* between these projects as well.

:::

:::tip Changing external storage only takes full effect after project reanalysis.

The selected storage is only updated once the project is reanalyzed. It is possible to change the selection without
reanalysis. In this case, the old selection will remain in use until the project is reanalyzed.
:::

### Configuring Uploads

If you are already uploading external analysis data in Teamscale via any of the described upload methods, no changes
are required on this end.
Once the external storage backend is configured in Teamscale, it will be recognized by Teamscale and uploads will be
redirected automatically.

### Replacing Existing Uploads

By default, an upload adds to what was uploaded before it: if you upload multiple coverage reports for the same commit and partition, the coverage of all uploads is accumulated.
If an upload should instead *replace* everything uploaded before it for that commit and partition, pass `replaceExistingUploads=true`.
This is useful to correct a faulty upload, for example when a build uploaded the report of the wrong test run.

```bash
curl --request POST --user bob:e434mikm35d --form "report=@coverage.xml" "http://teamscale-server:8080/api/vTEAMSCALE_VERSION/projects/myProject/external-analysis/session/auto-create/report?format=JACOCO&partition=Unit%20Tests&message=Corrected%20Unit%20Test%20Coverage&replaceExistingUploads=true"
```

The affected commit is then re-derived from the replacement upload alone.
To achieve that, Teamscale deletes the archives it previously stored for that commit and partition from the external storage.

:::warning Requires an external storage backend

`replaceExistingUploads` is only supported for projects that use an [external storage backend](#storing-analysis-results-in-an-external-storage-system).
Without one, the upload is rejected with HTTP 400.
:::

:::tip Pass it when the session is created

The parameter is evaluated when the upload session is created; that is, on an `auto-create` upload as shown above, or on the request that opens a session.
Passing it on an upload into an already opened session has no effect.
:::

:::tip Which uploads are replaced

Only uploads for the same commit and the same partition are replaced.
Other partitions of that commit and all other commits keep their data.
Uploads that identify their commit by timestamp instead of by revision are additionally scoped to their branch, so such an upload never replaces data of another branch.
:::

:::warning Identify the commit the same way in both uploads

A replacing upload only replaces earlier uploads that identify their commit the same way: either by revision, or by branch and timestamp.
This is a technical limitation: the identification determines where an upload is stored, and Teamscale looks for uploads to replace only where the replacing upload itself is stored.
So if you upload with `revision=<COMMIT_HASH>` and then upload the correction with only `t=<BRANCH>:<TIMESTAMP>` and `replaceExistingUploads=true`, the earlier upload is not found and stays in use, and the same happens the other way around.
Both uploads target the same commit, so it ends up with the accumulated data of both, which is what the flag was meant to avoid.
Teamscale does not warn about this.
:::

:::warning Replaced data is deleted

Teamscale deletes the replaced archives from the external storage.
You can only restore them through your storage system, for example with S3 versioning.
:::

If several uploads for the same commit and partition are still waiting to be stored when a newer one requests replacement, the older ones are dropped without being stored, since the replacing upload would remove them anyway.
Teamscale logs a warning if the flag is set for only some uploads of one commit and partition, because non-replacing uploads that precede a replacing one are dropped as well.

### Migrating Internally Stored Data to External Storage

If you have existing projects with internally stored external analysis data, you can migrate this data to an external
storage backend without reanalyzing the project.
Migration runs on your live production instance and only mirrors the data to external storage; the internally stored data is left in place.
Migration is started via REST API endpoints and runs in the
background while Teamscale remains usable. To use the migrated data, set up a second instance which configures
external storage normally. The second instance can already read the migrated data while the first instance is migrating.

:::tip Old Internally Stored Data is Not Deleted

The migration will not delete the internally stored data on the migrating instance.

:::

#### Prerequisites

Before triggering a migration, ensure the following on the production instance on which the migration will run:

* An external storage backend is configured in Teamscale (see [Configuring Teamscale to Store External Analysis Data in S3](#configuring-teamscale-to-store-external-analysis-data-in-s3))
* You have **both** "Edit Project" and "Perform External Uploads" permissions for the target project(s)
* Repository change retrievers have run at least once (to establish revision information)

:::tip No Reanalysis Required

The project does **not** need to be fully analyzed or in live analysis mode to migrate data. Migration works with the
existing revision information.

:::

:::warning One Migration Target at a Time

Only one migration target per project is allowed at a time. A migration is continuous and never completes on its own.
If a migration is already running for a project, attempting to migrate it to a different external storage backend will fail.
You must first [stop the active migration](#stopping-migration) before you can start a migration to a different backend.

:::

#### Triggering Migration

Migration is triggered via REST API. You can migrate all projects, specific projects, or a single project.

**Migrate all projects to an external storage backend:**

```bash
curl --request POST --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/<storage-backend-name>" \
  -H "Content-Type: application/json" \
  -d "[]"
```

**Migrate specific projects:**

```bash
curl --request POST --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/<storage-backend-name>" \
  -H "Content-Type: application/json" \
  -d '["project-id-1", "project-id-2"]'
```

**Migrate a single project:**

```bash
curl --request POST --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/<storage-backend-name>/<project-id>"
```

**Parameters:**

* `<storage-backend-name>`: Name of the configured external storage backend (as shown in the project settings)
* `<project-id>`: Project ID, visible in the Teamscale URL when viewing the project or in the *Projects* view
* `<username>:<accesskey>`: Your Teamscale username and [access key](../../glossary/#access-key)

:::tip Interactive API Documentation

For detailed parameter descriptions and additional options, see the [REST API documentation](../../reference/rest-api/#available-rest-services).

:::

#### What Happens During Migration

When you trigger a migration:

1. Teamscale identifies all analysis data currently stored in the internal database for the selected project(s).
2. Data is grouped by commit and staged for upload.
3. The data is transferred to the external storage backend using the same upload mechanisms as during live operation.
4. **Zero Downtime:** Original internal data remains accessible during the process, and Teamscale remains fully functional.

:::tip Reading while migrating

A second Teamscale instance can analyze the migrated data incrementally while the first instance is still migrating.
We recommend using a "shadow" instance for this, which is typically used when updating to a new major Teamscale version.

:::

:::tip Uploading New Data During Migration

You can continue uploading new external analysis data while migration is running. New uploads will be handled automatically
and do not interfere with the migration process.

:::

:::warning Migrations Are Unaffected by Configuration Changes

Migrations create a snapshot of the external storage backend configuration when they start and will keep using this snapshot.
Hence, any changes to the external storage migration will not affect running migrations.

:::

#### Monitoring Progress

**Check migration progress for a specific project:**

```bash
curl --request GET --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/progress/<project-id>"
```

This returns counts for `migrated`, `pending`, and `errors`.

**Check progress for all projects currently being migrated:**

```bash
curl --request GET --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/progress"
```

#### Stopping Migration

If you need to stop the continuous migration for a project, use the DELETE endpoint:

```bash
curl --request DELETE --user <username>:<accesskey> \
  "http://localhost:8080/api/vTEAMSCALE_VERSION/external-storage/migrate-uploads/<project-id>"
```

:::info Restarting a Migration

To migrate a project that was migrated before, stop its migration if it still runs, clear the external storage, and add `?reset=true` to the request that starts the migration.
This migrates all uploads of the project again and also replaces a running migration to a different backend.
Without `?reset=true`, Teamscale rejects the request for such a project, and a migration of all projects skips it.

If Teamscale restarts while a migration runs, the migration continues after the restart.

:::

:::warning Uploads Being Processed May Still Complete

Stopping a migration removes the project from the migration index, prevents future migration triggers from being scheduled, and discards the uploads that are still pending.
Migration triggers that are already queued skip the project, and a migration trigger that is already running stops before it migrates its next uploads.
Uploads being processed at the time you stop the migration may still complete.

:::

#### Switching to External Storage

Migrated data can be read on a new Teamscale instance by importing the instance backup from the migrating instance.
This will automatically exclude external uploads from the backup to avoid duplicate data imports on the importing instance.
The new instance will already start reading and analyzing available data from external storage while the old instance is migrating.
We recommend combining the switch with a regular major update, as this requires setting up a fresh instance with a backup import anyway.

:::info Migrated Data is Ignored By Migrating Instance

The migrating instance will ignore the migrated data from external storage to avoid rollbacks.
To read the data, set up a fresh instance using a backup from the migrating instance.

:::

:::warning Enabling Migration Excludes External Uploads From Backup

The global backup export will automatically exclude external uploads from the backup for all projects which are currently migrating.
Nightly backup exports via the "Automatic Backup" option are not affected by this and will still export all data.
Hence, use a manual backup export to set up a new instance for reading from external storage.

:::

### Disabling External Storage Backends

To disable external storage, select "Internal Storage" for the projects which should no longer use the system. This
change takes effect on the next project reanalysis.

:::tip

Resetting the Teamscale configuration does not delete any data in the external storage system.
:::
