Setting up Test Gap Analysis on a .NET Desktop Application (Pinta)

As a developer or tester of a .NET desktop application, you may use Teamscale's Test Gap Analysis (TGA) to improve your testing effectiveness. This tutorial leads you through the setup of TGA on Pinta (opens new window), an open source graphics editor.

• Step 1: Installing Pinta
• Step 2: Installing the Teamscale Ephemeral Profiler
• Step 3: Uploading Coverage to Teamscale
  • Step 3.1: Preparing Teamscale
  • Step 3.2: Linking the Version Under Test to a Source Code Revision
  • Step 3.3: Uploading Coverage
• Step 4: Analyzing Coverage and Test Gaps
  • Step 4.1: Test Gaps at File Level
  • Step 4.2: Test Gaps at Method Level

Step 1: Installing Pinta

As a prerequisite for running Pinta, install a .NET Framework 4.5 (or newer) and GTK# for Windows (opens new window).

Next, place the Pinta binaries somewhere on your local machine and the corresponding PDB (symbol) files in C:\pinta_pdbs. To obtain these files, either compile the Pinta sourcecode yourself or download our demo bundle:

TGA Demo Bundle with Pinta 1.6 binaries and PDBs

Finally, start Pinta, to ensure that the installation was successful.

Troubleshooting: Pinta Fails to Start

Ensure there's only one version of GTK# installed on your system, otherwise starting Pinta might fail due to DLL loading problems.

Step 2: Installing the Teamscale Ephemeral Profiler

Download the latest release of the Teamscale Ephemeral Profiler (opens new window) and unpack it to C:\teamscale_dotnet_profiler.

To register the profiler, multiple environment variables have to be set. This can be accomplished either by creating a launch script, or by assigning the environment variables system-wide in the Control Panel. For an example on creating a launch script, check PintaWithProfiling.bat within the demo bundle. If you want to set the environment variables system-wide, navigate to Control Panel > System > Advanced system settings > Environment Variables and enter the following lines:

• COR_ENABLE_PROFILING: 1
• COR_PROFILER: {DD0A1BB6-11CE-11DD-8EE8-3F9E55D89593}
• COR_PROFILER_PATH: C:\teamscale_dotnet_profiler\Profiler32.dll (or Profiler64.dll, depending on the bitness of the Pinta process)

Setting the COR_ENABLE_PROFILING variable enables profiling on .NET processes. The COR_PROFILER variable provides the GUID of the Teamscale Ephemeral Profiler, which the .NET Runtime uses to verify the profiler DLL on load. Finally, the COR_PROFILER_PATH tells the .NET Runtime where to find the profiler binaries. Profiling a .NET process requires a different version of the profiler, depending on whether the process is a 32 or 64 bit process. To determine the bitness of the Pinta process, follow our guide on determining application bitness on Windows (opens new window).

Next, we configure the profiler itself. Rename C:\teamscale_dotnet_profiler\Profiler.example.yml to C:\teamscale_dotnet_profiler\Profiler.yml, then edit the file.

Adjust the global configuration section, i.e., the first entry below match, as follows:

match:
  # global configuration section
  - profiler:
      enabled: false
      targetdir: C:\pinta_coverage
      upload_daemon: false

This globally disables both the profiler, to avoid profiling unnecessary processes, and the automatic upload of coverage, which we will configure later. Moreover, it instructs the profiler to write trace files to C:\pinta_coverage.

Permissions

Ensure that C:\pinta_coverage exists and that the user running Pinta.exe can write to it. Otherwise, profiling the process will fail.

Next, add a process-specific configuration sections below the global one (delete any such sections that exist in the template), to enable the profiler specifically for the Pinta executable:

match:
  # global configuration section
  - ...
  # process-specific configuration section
  - executableName: Pinta.exe
    profiler:
      enabled: true

To test your setup, start Pinta, do something (e.g., create a new image and draw a Picasso), and closing it again. The target directory C:\pinta_coverage should now contain a coverage report named like coverage_20190307_1352510131.txt (the date and time parts of the file name will differ) with some content.

Reduce Performance Impact

Profiling processes always incurs a performance impact on the processes. The Teamscale Ephemeral Profiler minimizes this impact, but cannot completely avoid it. To minimize the overall impact, ensure you always configure the minimal set of processes (executables).

Step 3: Uploading Coverage to Teamscale

The Teamscale Ephemeral Profiler comes with the ability to upload coverage directly to a Teamscale instance. During the upload, the profiler maps the recorded coverage to the source code of the application under test, using the respective PDB files.

Assumption: Reachability of Teamscale

This guide assumes that the Teamscale server is directly reachable from the test environment via HTTP(S). Alternatively, the profiler can upload reports to a file share (opens new window), where they may be picked up and forwarded from another machine.

Step 3.1: Preparing Teamscale

In Teamscale, go to Project Configuration and click New project.

Name the new project pinta and choose the analysis profile C# Default.

In the Branching Configuration, change all start dates to 2014-03-25.

Click Source Code Repository and select Git. Create a new account named Pinta GitHub with the URI https://github.com/PintaProject/Pinta.git and without credentials. Set the Default branch name to release-1.6. Clear the Start revision as we have already set it for the whole project in the Branching Configuration.

Start revision and Branching Configuration interact

If you create a project with more than one source code connector (e.g. to integrate multiple Git repositories into one Teamscale project), you can control how much history is analyzed by Teamscale centrally via the Branching Configuration. It applies to all connectors.

If you set a Start revision in addition, the more recent of the two dates is applied to this connector.

Click Create project.

Step 3.2: Linking the Version Under Test to a Source Code Revision

Create the file C:\revision.txt and set the file's content to revision: 6f987872ff93854a841bce1e71987e4d70c82b86. This commit ID identifies the source code corresponding to the Pinta 1.6 release, which we want to link the coverage information to.

Keep the Version Under Test and revision.txt in Sync

If you change the version under test, you also need to adjust the revision.txt accordingly. We recommend to automate this in real test environments.

Step 3.3: Uploading Coverage

Edit C:\teamscale_dotnet_profiler\Profiler.yml again and change the global configuration section to match the following:

match:
  # global configuration section
  - profiler:
      enabled: false
      targetdir: C:\pinta_coverage
      upload_daemon: true
    uploader:
      teamscale:
        url: http://localhost:8080 # use your URL
        username: admin # use your username
        accessKey: u7a9abc32r45r2uiig3vvv # use your access key
        project: pinta
        partition: Manual Tests
      pdbDirectory: C:\pinta_pdbs
      revisionFile: C:\revision.txt
      assemblyPatterns:
        include:
          - Pinta.*
  ...

This globally enables and configures automatic uploading of coverage data.

Use the URL and credentials (username and access key) of your Teamscale instance. The project option contains the ID of your Teamscale project such that the coverage data is sent to this project. The partition is an arbitrary label that you may use to distinguish coverage source, such as different test stages. Teamscale merges all coverage in the same partition and allows you to inspect coverage in different partitions individually (or in any combination).

During the upload, the Teamscale Ephemeral Profiler maps the coverage data back to a particular version of the source code, using the PDB files and the revision ID from the revision.txt. It will do so for all the assemblies of Pinta (whose names match the regular expression Pinta.*).

To test your setup, start Pinta, do something (e.g., create a new image and draw a Picasso), and close it again. The profiler automatically schedules uploading of coverage data from terminated processes every five minutes. To verify this worked, in Teamscale, go to Activity. Once the data arrived and was processes successfully, the entry on top of the timeline represents the coverage upload.

Environment variables

Note that if you did not set the environment variables system-wide, but inside a launch script, ensure that you are starting Pinta using the launch script (i.e. PintaWithProfiling.bat) instead of running Pinta.exe directly. Otherwise this will result in a wrong configuration of the environment variables.

Permissions

Ensure that the user running Pinta.exe has read access to C:\pinta_pdbs. Otherwise, processing the coverage will fail.

Upload Interval

You may configure the upload interval in the Profiler.yml file, using the uploadIntervalInMinutes option.

Reduce Performance Impact

Mapping coverage data to source code takes some processing time that adds to your overall test execution time. To avoid wasting time on unnecessary mapping, ensure you always configure the minimal set of assemblies, i.e., exactly the assemblies containing the code that you want to identify test gaps in.

Step 4: Analyzing Coverage and Test Gaps

In Teamscale, go to Test Gaps > Files and click Baseline (above the Files table) to open the Set baseline for code changes... dialog. Select Baseline > Add Baseline to create a baseline named Pinta 1.5 at revision 26c5a365, which is the first analyzed commit in the release-1.6 branch after the the Pinta 1.5 release (opens new window). Select the newly created baseline and click OK.

Step 4.1: Test Gaps at File Level

In Teamscale, go to Test Gaps and choose the Pinta 1.5 baseline at the top.

The table on the page shows the file-system structure of the Pinta project, with Test Gaps, Execution, and Churn data aggregated for each file or directory. You may drill down in this perspective to check what exactly has been changed or executed and where the test gaps are.

Hint

The Execution and Test Gap data may look different for you, depending on the actions you performed in Pinta.

Step 4.2: Test Gaps at Method Level

In Teamscale, go to Dashboards. In the right sidebar click Add dashboard and then choose Test Gap Treemap from the list of widgets. Finally, click Save Dashboard and name the Dashboard Pinta Test Gaps 1.5 -> 1.6.

Move the mouse over the Test-Gap Treemap and click on the icon that appears in the upper left, to edit the widget. As the Baseline, choose the Pinta 1.5 baseline. Make sure that Manual Tests is selected under Coverage Sources, for the widget to consider the coverage you uploaded earlier. Confirm the dialog.

The widget now shows the code changes between version 1.5 and 1.6 in color. The red and yellow blocks represent methods that have been changed, but where not covered by tests, i.e., test gaps. Hover the mouse over the blocks in the treemap to see details in a tooltip. Click on the blocks the drill into the data.

Hint

The treemap may look different for you, depending on the actions you performed in Pinta.