How to Handle Backups (Import, Export, Automatic Backups)
Backups in Teamscale are very helpful when upgrading to a new Teamscale version or to mitigate the risk of potential data loss.
Automatic Backups
We highly recommend automatic backups to be enabled for all production use of Teamscale. See the respective section below on how to configure automatic backups.
Exporting Backups
Depending on your needs (e.g., during an update), you can export only global data (like users, groups, settings, permissions, dashboards), project data (e.g., tolerations, false positives, baselines and external findings), or both. Furthermore, you can choose to only export project data for some rather than all of your projects.
To create a backup, first navigate to Admin > Backup. Here you have two options to initiate the backup:
When you choose Export Backup, your browser will download a single backup ZIP and you can store it locally.
Alternatively, you can provide a file-system path (or S3 URI) to let Teamscale store your backup there. This way is often preferable if the backup ZIP would be too large to comfortably download.
Rescue backup on startup
Under certain error conditions (e.g. database corruption) the above mechanism may not work anymore for exporting backups. In this case the command line argument -b <PATH-TO-BACKUP-FILE> passed to the teamscale.sh or .bat file can be used to dump a rescue backup during the startup of Teamscale.
Importing backups
You have three options to import a backup:
The first option is the tab Upload Backup File. Your browser will then let you select a single backup ZIP which you have downloaded previously. Clicking on the Upload Backup button will initiate the import.
The second option is the tab Load Backup from other instance. In this case you have to provide the URL and the credentials of another Teamscale instance. The URL of this other instance has to be reachable from your current one. After clicking on the Import Backup button, the remote Teamscale instance will export a backup which then will be imported on your current instance.
The third option is to provide a file-system path (or S3 URI) to a previously exported backup on the tab Read Backup From File or URL. Then, choose Import Backup to let Teamscale fetch the backup and import it.
Automated backups
For all production uses we highly recommend configuring automated backups. Once configured, Teamscale will regularly perform an automated backup export. This ensures that you can always resort to an up-to-date backup in case of data loss.
Default Automatic Backup Settings
Automatic backups are enabled automatically on the first start of Teamscale using default settings for the backup storage location and for how many backup files will be kept. We recommend revisiting the default settings of automatic backups to make sure that they are in line with your preferences.
To configure automated backups, open the Admin perspective and navigate to Settings > Automatic Backup. 
Here you can configure the file name pattern of the backup to be created. Several placeholders can be used to better identify a backup file as described in the description of the settings page. It is recommended to use the %t placeholder in the file name for reflecting the creation time in the file name.
The "Backup schedule" setting allows to define the frequency of the automatic backups. This setting also defines whether the automatic backup is enabled at all (leave empty to disable). We highly recommend a nightly schedule. To ensure that the backups created to not exceed the disk space available, options exist on how many backups should be preserved for a given interval. For example, you can configure Teamscale to keep daily backups for the last 7 days, weekly backups for the last 5 weeks and monthly backups for 3 months. Teamscale will regularly delete all other backups to free up disk space.
Backup to/from Amazon AWS S3
All aforementioned backup options (including automated backup) also work to/from AWS S3 buckets (or with S3-compatible storage solutions). Instead of specifying a file-system path, you simply specify an S3 URI, e.g., s3://bucket/key or s3+https://acesskey:secretkey@s3.example.com/bucket/path . Rotation settings of automatic backups are also applied to backups stored in S3.
You have two options to specify the path to an S3 bucket: s3 & s3+http(s) URLS.
S3 URLs
Here, the format of the URI is as follows:
s3://<bucket>/<path>To use this URL format, you must set up the machine's AWS environment outside of Teamscale. At a bare minimum, you need to define the access key ID, the access key and the bucket region.
S3+HTTP(S) URLS
In this case, the format of the URI is as follows:
(s3+http|s3+https)://[<access key>:<secret key>@]<hostname>[:<port>]/<bucket>/<path>The S3 bucket must be specified as part of the URI's path, not the hostname. In other words, you have to use path-style access rather than virtual-host-style access.
If either access key or secret key contain a slash (/), it must be percent-escaped as %2F.
Instead of providing your access and secret key directly in the URI, it is also possible to store them under Admin > External Credentials. To do so, create an external credential entry whose ID and URL both match your chosen hostname and port: http(s)://<hostname>[:<port>]. The entry's Username and Password then are your S3 access key and secret key, respectively.
Storage snapshot backups
Teamscale can be configured to automatically create snapshots of its database.
Configuring Automatic Snapshots
The backups created by Teamscale contain all user generated data (e.g. configurations, architectures, dashboards, and tolerated findings), and uploads (such as coverage data). These backups are useful when migrating to a new version or recovering data that was mistakenly deleted. However, restarting an instance from such a backup requires a reanalysis, which can take some time and might be unsuitable for crash recovery, when fast recovery time is more important.
This gap is filled by storage snapshot backups, which store a snapshot of the entire storage system in one backup file, which has a size that is almost as big as the storage directory. These storage snapshot backups can be created manually in the Backup view of the Admin perspective. It is recommended to configure a regular schedule for creating storage snapshots in the Settings > Automatic Backup view, in the Automatic Storage System Snapshots section. The backup location supports the same placeholders and URI schemes as the normal backup, so storing snapshots in S3 is possible.
Snapshot Support
Note that storage snapshot backups are only supported by these database types: RocksDB, In-Memory (see also Configuring Teamscale on how to configure the database type).
Recovering using a Storage Snapshot
To recover from a crash and start a Teamscale instance from the storage snapshot, you have to start the instance with a specific property. For this, add the following parameter to the jvm.properties config file.
-Dcom.teamscale.snapshot-backup.load-location=LOCATIONPlease note that the import will only be performed, if the storage system is completely empty, to prevent you from accidentally destroying a live instance. In this case, a warning will be logged and the existing instance will be started without any import. Additionally, the version of Teamscale must be the same as the version that was used to create the snapshot. You can not use snapshots during a migration to a newer version of Teamscale.
When working with storage snapshot backups, please consider these general guidelines:
Backups should be stored on a separate machine (or even data center) to prevent data loss in case of disk failures or environmental hazards.
The restore procedure should be tested regularly, both to ensure that all data is available and the steps are known and also to get an idea of how long the restore procedure takes.
Also think about backup procedures for the operating system and hardware for a fallback machine, as in case of a hardware failure, you will also need a new machine to start from.
Think about procedures to switch URLs to a new machine in case of hardware failures, such as load balancers or suitable DNS update strategies.