---
url: /howto/monitoring-teamscale.md
---

# How to Monitor a Teamscale Instance

To check and monitor the state of a Teamscale instance, the following options are available:

## Monitoring via Web UI

Teamscale provides many helpful metrics via the **System Information** view
in the **System** perspective. In addition, the logs (e.g., **Worker Log**) can be helpful for diagnosis.

## API Endpoint for Nagios

The URLs `api/health-check` and `api/health-metrics` provide check results
and metrics in the [Nagios format](https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/pluginapi.html).
This can be used with Nagios or compatible solutions, such as Sensu, to monitor the current health
status of Teamscale. To integrate on the command-line, see the
`monitoring` directory in your Teamscale distribution.

:::details Exposed Metrics
General information about the instance

* State of the scheduler
* If the instance has run out of memory
* If the instance is running out of disk space
* If the Java VM is running out of memory
* If the license is valid or outdated
* If the server certificate is valid
* The version of Teamscale
* Number of workers
* The load of the worker
* Number of licensed user
* Number of users
* Number of active users in specific time frames
* Number of committers in specific time frames
* Number of projects
  :::

## API Endpoint for Prometheus

The URL `api/monitoring/prometheus` exposes various metrics of Teamscale in the
[Prometheus format](https://prometheus.io/docs/instrumenting/exposition_formats/).
[Our reference documentation](../../reference/admin-settings/monitoring/#prometheus-service) contains more information on how to enable and configure Teamscale for use with Prometheus.

:::details Exposed Metrics
General information about the instance

* Name of the instance
* Name of the process
* The version of Teamscale
* If the instance is in shadow mode
* State of the scheduler
* Number of workers
* The load of the workers
* The current runtime of each worker
* The size of the job queue
* Days left until license expires
* Number of licensed user
* Number of users
* Number of active users in specific time frames
* Number of committers in specific time frames
* Number of projects
* CPU load
* Number of logical CPU cores
* Size of the RAM
* Amount of used RAM
* Detailed state of the Java VM
  * Used RAM
  * Number of threads
  * Time spent in garbage collection
  * and more
* Statistics of the Internal String Abbreviator Cache
* Statistics on service calls
* Directory size of the working directory, storage directory, (git) repository directory and temp directory used by Teamscale
* Amount of critical system events
* Number of successful automatic backup runs
* Timestamp of the last successful automatic backup
* Usage of the process pools of integrated tools

Metrics for each project.

* Primary public ID of the project
* Analysis state of the project
* Number of connectors in specific states
* Number of files
* Number of lines of code
* Number of commits in specific states
* Number of the different log entries
* Number of rolled back commits within the last 24 hours
* Duration of rollback executions
* Number of currently postponed rollbacks
* Whether the project is paused
* Statistics on pre-commit usage

Storage performance metrics. These metrics are disabled by default because they
are expensive to collect and are only useful for debugging.
To enable these metrics set the flag `-Dcom.teamscale.storage-metrics.enabled=true`.
More information how to set a flag can be seen [here](../../reference/administration-ts-installation/index.md#jvm-arguments).

* Number of opening operations for a store
* Number of storage operations
* Number of overall affected keys in storage operations
* Number of overall bytes in keys
* Number of overall bytes in values
* Duration in milliseconds of storage operations
  :::

## Probe Endpoints for Kubernetes

Teamscale serves one endpoint per Kubernetes probe type.
All three are unauthenticated by design, so that they also work on instances that otherwise permit only SSO/SAML logins, where a kubelet could never obtain a session.

### Pod Spec

```yaml
spec:
  containers:
  - name: teamscale
    image: cqse/teamscale:<version>
    ports:
    - name: http
      containerPort: 8080

    startupProbe:
      httpGet:
        path: /api/probes/startup
        port: http
      periodSeconds: 10
      failureThreshold: 90

    livenessProbe:
      httpGet:
        path: /api/probes/liveness
        port: http
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3

    readinessProbe:
      httpGet:
        path: /api/probes/readiness
        port: http
      periodSeconds: 10
      failureThreshold: 2
```

The startup probe is what lets you budget minutes for bootstrapping without weakening the liveness timeout, so configure it before the other two on any instance with a sizeable storage system.

### Paths

A configured `server.urlprefix` applies to the probe paths as well, so the prefix `teamscale` makes the liveness path `/teamscale/api/probes/liveness`.

All three endpoints belong to the public, versioned [REST API](../../reference/rest-api/) and are equally reachable as `/api/vTEAMSCALE_VERSION/probes/liveness`.
The unversioned path is the better choice in a manifest, because the probes return no data model that a version bump could change.

## Forwarding Teamscale Logs to Splunk

Teamscale uses the Log4J logging framework and provides support for forwarding the generated logs to a Splunk server. Log forwarding can be configured using the [Splunk logging for Java](https://dev.splunk.com/enterprise/docs/devtools/java/logging-java) integration and Teamscale fully supports HTTP Event Collector (recommended) and TCP data inputs. See the default `log4j2.yaml` Log4J configuration file in the Teamscale distribution Zip for an example configuration. For further configuration options refer to the official [How to use Splunk logging for Java](https://dev.splunk.com/enterprise/docs/devtools/java/logging-java/howtouseloggingjava/enableloghttpjava/#2-Enable-HTTP-Event-Collector) page. To reduce load on the Splunk server, consider adjusting the `batch_interval`, `batch_size_bytes`, or `batch_size_count` of the Log4J appender for Splunk to reduce the frequency of log forwarding.
